Ouyang's Blog

Working on ML Infra

0%

Tools cheatsheet

Posted on Edited on In Disqus:

CI: clang and yapf

  • clang-format -style=file -i tensorflow/core/kernels/ResourceSparseApplyAdagradV2.*
  • yapf -i --style pep8 tensorflow/python/test_abs.py
  • cpplint --linelength=80 --counting=detailed --filter=-build/namespaces --root=src $(find src/pytorch_ops -name “.h” -or -name “.cc”)
  • cpplint --linelength=80 --counting=detailed --root=src $(find src -path “src/pytorch_ops” -prune -name “.h” -or -name “.cc”)

GDB and nm

  • nm -D BINARY // Only show BINARY (object file, static/dynamic library, executable) dynamic section’s symbol: T/t means golbal/local symbol. U means undefined symbol.
  • nm -Du BINARY // Only show undefined symbol
  • c++filt // Transform symbols in so to human readable form

Performance Monitoring Tools

  • Program Optimization: Gprof, Perf, VTune
  • Program Debugging: Valgrind, Sanitizers
  • Advanced Analysis: Pin, Contech

Linux

  • Check ip: curl ip.gs
  • Syspend: systemctl suspend | systemctl hibernate
  • Show file size: ls -lht
  • nohup docker pull tensorflow/- tensorflow:nightly-custom-op-gpu-ubuntu16 &> docker_pull.out &
  • tar -xvf etc.tar
  • CRLF: dos2unix < myfile.txt | cmp - myfile.txt
  • find . -name “*.so” -exec nm {} ; | c++filt | grep Dense
  • scp -r username@192.168.3.33:c:/Users/username/thisComputer/project ./project
  • ln -s {source file} {distination file}

    bazel

  • sudo apt update && sudo apt install graphviz xdot
  • xdot <(bazel query –notool_deps –noimplicit_deps “deps(//main:hello-world)”

–output graph)

  • bazel –distdir
  • bazel query –notool_deps –noimplicit_deps “deps(//tensorflow/core/kernels:training_ops)” –output graph
  • dot -Tpng -T svg graph.log -o graph.png

Tmux

  • Attach: tmux a

  • exit | ctrl + d

  • tmux new -s

  • tmux attach -t try

  • Window Operation

    • Ctrl+B c // Create a new window
    • Ctrl+B & // Close current window
    • Ctrl+B p // Switch to previous window
    • Ctrl+B n // Switch to next window
    • Ctrl+B {window number} // Switch to corresponding window (For example if window number is one, then Ctrl-b 1)
    • Ctrl+B , // Rename current window

  • Panel Operation

    • Ctrl+B % // Vertically divide Terminal (left and right)
    • Ctrl+B “ // Horizontally divide Terminal
    • Ctrl+B {arrow} // Switch to different panel
    • Ctrl+B x // Close current panel
    • Ctrl+B q // Show panel’s serial number

  • Session Operation

    • Ctrl+B s // List all sessions
    • Ctrl+B d // Detach current session(runing in background)

  • search mode

    • Ctrl-b:set-window-option -g mode-keys vi
    • Ctrl-b [

PIP and Python site-packages

  • pip install -i https://pypi.tuna.tsinghua.edu.cn/simple
  • or add the following lines to ~/.pip/pip.conf
  • 1
    2
    [global]
    index-url = https://pypi.tuna.tsinghua.edu.cn/simple
  • Python
    1
    2
    3
    4
    5
    6
    7
    8
    9
    dir(instance)
    instance.__dir__()
    instance.__dict__

    import tensorflow as tf
    tf.sysconfig.get_include()
    '/usr/local/lib/python3.6/site-packages/tensorflow/include'
    tf.sysconfig.get_lib()
    '/usr/local/lib/python3.6/site-packages/tensorflow'

Shell

  • set -e: If there is an error when executing a command, then exit immediately.
  • set -x: Show the command and args before executing the command. For debug purpose.
  • If condition in Dockerfile:
    1
    2
    3
    FROM centos:7
    ARG arg
    RUN if [[ -z "$arg" ]] ; then echo Argument not provided ; else echo Argument is $arg ; fi

NPM

  • npm install –package-lock-only

Ubuntu Container Init Example