Software Engineering

Software Engineering

Principles #

  • Start simple and iterate
  • Make it hard to do the wrong thing
  • Make your systems fail gracefully
  • Put checks, asserts, debug output everywhere
  • Measure before you optimize

Documents #

Design and Decision #

Should contain:

  • Author, reviewers, date
  • Intro, abstract
  • Goals and scope
  • Proposed solution/decision
  • Implementation steps

Can contain:

  • Problem analysis
  • Context
  • Non-goals
  • Existing solution
  • Alternative solutions considered
  • Cost
  • Evaluation
  • Resources
  • Open questions, to do, notes
  • Future enhancements

Testing/Experimentation Plan #

Write before the experiment:

  • Author, reviewers, date
  • Intro, abstract
  • Goals and desired outcome
  • Data to collect
  • Equipment and prerequisites needed
  • Procedure

Write after the experiment:

  • Execution log
  • Deviations from procedure
  • Results, analysis
  • Discussion

Inline Documentation #

Inline code documentation on classes/methods/data needs the following info:

  • Cardinality
  • Input/output examples
  • Run time, O(…)
  • Data storage, ownership
  • Life cycle
  • Blocking vs. async
  • Naming

Pull requests #

  • As small as possible
  • Squash-merge only
  • First submit - first merge
  • Never merge a PR manually on your own machine
  • Only the person who submitted may merge

Linting & Project Setup #

Python Trinity #

flake8
pytype
black --check --line-length 99 --exclude '/(\.git|\.pytype|venv|venv_test)/' .

setup.cfg

[flake8]
exclude = venv/*,
    .git,
    __pycache__,
    *_test.py

# conflicts with black
ignore = W503, E203, E501

[pytype]
exclude =
    **/*_test.py
    venv/*
inputs = **/*.py
python_version = 3.6
pythonpath = .
keep_going = True

Clang-Format #

#!/bin/zsh

set -xe

# set working dir to repo directory
cd "${0%/*}/.."

clang-format --version
clang-format -style=file -i \
    src/**/*.cc \
    src/**/*.hs

.clang-format

---
Language:            Cpp
BasedOnStyle:        Google
IncludeBlocks:       Preserve
BreakStringLiterals: false
ColumnLimit:         99

C++ Build #

.bazelrc

# force using clang-x
build --action_env=CC=/usr/bin/clang-x
build --action_env=CXX=/usr/bin/clang++-x

build --workspace_status_command=status.sh
# COPTS.bzl
COPTS = [
    "-std=c++14",
    "-Wall",
    "-Werror",
    "-Wextra",
    "-Wpedantic",
    "-Wshadow",
    "-Wsign-compare",
    "-Wunreachable-code",
    "-g",
    "-iquotesrc/",
]

# BUILD
load("//src:COPTS.bzl", "COPTS")

cc_binary(
    copts = COPTS,
    ...
)

Tests #

C++ Test structure #

*_test.cc

TEST(NameOfClassOrFnTest, Simple) {
  ...
}

TEST(NameOfClassOrFnTest, DoesSomething) {
  ...
}

TEST(NameOfClassOrFnTest, DoesSomethingElse) {
  ...
}

Python Test structure #

*_test.py

import unittest
from unittest.mock import MagicMock, Mock, patch
from truth.truth import AssertThat

class ClassNameTest(unittest.TestCase):
    def setUp(self):
        self.x = MagicMock(spec=ClassOrFunctionBeingTested)

    def testSomeMethod_simple(self):
        ...

    def testSomeMethod_handlesException(self):
        ...

    def testSomeMethod_doesSomething(self):
        ...

Run using

python -m unittest discover -p '*_test.py'

python -m unittest my_module.my_submodule_test
python -m unittest my_module.my_submodule_test.ClassOrFunctionNameTest
python -m unittest my_module.my_submodule_test.ClassOrFunctionNameTest.testSomeMethod_simple

Various #

Equivalent Dockerignore/Gitignore #

.gitignore

# dependencies
node_modules
.pnp
.pnp.js

# misc
.vscode
.DS_Store

.dockerignore

# dependencies
**/node_modules
**/.pnp
**/.pnp.js

# misc
**/.vscode
**/.DS_Store
**/Dockerfile
**/.dockerignore
**/docker-compose.yml
**/README.md

Creating a private fork on Github #

git clone git@github.com:org/repo.git
cd repo
git remote rm origin
git remote add origin git@github.com:private-org/repo.git
git remote add upstream git@github.com:org/repo.git
git push --all
git push --tags
git fetch
git checkout -b upstream-master upstream/master
git checkout master
git checkout -b my-master
git push --set-upstream origin my-master

Updating from upstream:

git checkout upstream-master
git pull
git checkout -b master origin/master # first time only
git checkout master
git merge upstream-master
git checkout my-master
git merge upstream-master
git push --all

Ruin somebodys day (in C) #

Sneak into his headers:

# define struct union
# define else

Cmake: print all variables #

get_cmake_property(_variableNames VARIABLES)
foreach (_variableName ${_variableNames})
    message(STATUS "${_variableName}=${${_variableName}}")
endforeach()

AWS ClouWatch Insights #

# show latest logs from a certain stream "mystream"
fields @logStream, @timestamp, @message
| filter @logStream like /i-[0-9a-z]+-mystream/
| sort @timestamp desc
| limit 2000

# syslog without "awslogs-nanny.sh"
fields @timestamp, @message
| filter @logStream like /i-[0-9a-z]+-syslog/
| filter @message like /^((?!awslogs-nanny\.sh).)*$/
| sort @timestamp desc
| limit 200

# only lines with "SomeMessage"
fields @timestamp, @message
| filter @logStream like /i-[0-9a-z]+-syslog/
| filter @message like /SomeMessage/
| sort @timestamp desc
| limit 2000