Ouyang's Blog

Docker notes

Posted on 2021-03-08 Edited on 2022-02-16 In Tools Disqus:

Run and exec

docker run --ipc=host --gpus all --entrypoint=/bin/bash -it -d -v /home/name:/home/name -e TZ=Asia/Beijing --name ouyang tensorflow/tensorflow:2.4.1-gpu
run flags:
- --pid=host: Use it if you want your container to share the host’s process namespace, basically allowing processes within the container to see all of the processes on the system.
- --ipc=private|shareable|host|container:name-or-id: IPC namespace provides separation of named shared memory segments, semaphores and message queues. Shared memory segments is commenly used by databases and C/OpenMPI, C++/using boost libraries and etc.
docker exec -it ${container_name} /bin/bash
docker exec -it -w /home/name ouyang sh -c “tmux a” /bin/bash

System

docker system df
docker system df -v
sudo service docker start/stop

TensorFlow HashTable Operator Source note

Posted on 2021-02-24 Edited on 2022-02-16 In TensorFlow Disqus:

Structure of TensorFlow HashTable source code

core/kernels/lookup_table_op.h

class LookupTableOp:
- This class inherits from the OpKernel class and supports different table implementations specified by the Container template.
- It has a member PersistentTensor table_handle_. This member is stored in the OpKernelContext by ctx->allocate_persistent.
Container must be inherited from LookupInterface. Container example: MutableDenseHashTable, MutableHashTableOfTensors and etc.
class HashTable: This class inherits from InitializableLookupTable class and wraps an flat_hash_map, where the key and value data type is specified. Once the table is marked as initialized it becomes read-only.

core/kernels/lookup_table_op.cc

LookupTableOpKernel: This class inherits from the OpKernel.
It will make sure the table_handle_ will be the first input for ops such as LookupTableFind, LookupTableInsert, etc.

This class will be inherited by operations like Remove, Insert, Find, Size, Export, Import and etc. For example:

1
2
3

class LookupTableOpKernel : public OpKernel;
class LookupTableFindOp : public LookupTableOpKernel;
REGISTER_KERNEL_BUILDER(Name("LookupTableFindV2").LookupTableFindOp);

Here are some examples of hashtable kernel builder:

1
2
3

REGISTER_KERNEL_BUILDER(Name("HashTableV2"), LookupTableOp<lookup::HashTable<key_dtype, value_dtype>, key_dtype, value_dtype>);
REGISTER_KERNEL_BUILDER(Name("MutableHashTableV2"), LookupTableOp<lookup::MutableHashTableOfScalars<key_dtype, value_dtype>, key_dtype, value_dtype>);
REGISTER_KERNEL_BUILDER(Name("MutableHashTableOfTensorsV2"), LookupTableOp<lookup::MutableHashTableOfTensors<key_dtype, value_dtype>, key_dtype, value_dtype>);

CMake notes

Posted on 2021-02-01 Edited on 2022-02-16 In Languages Disqus:

Why “file(GLOB xxx)” is evil

It will cause inconspicuous problem during teamwork.
A adds a new file and does not change CMakeLists.txt
B does a git pull, and since there have been no modifications to the CMakeLists.txt, CMake isn’t run again and B causes linker errors when compiling.
Reference: https://stackoverflow.com/questions/32411963/why-is-cmake-file-glob-evil

About find_package

Check local modules: cmake –help-module-list
Manual for a specific module: cmake –help-module FindCUDA

Read more »

Git Cheatsheet

Posted on 2021-01-15 Edited on 2022-02-16 In Tools Disqus:

Good git resources

Git Move files from one repository to another while preserving git history

mkdir merge_repo && cd merge_repo
git clone --branch master --origin origin --progress -v git@github:xxx/projectA.git
cd projectA
git remote rm origin
git filter-branch --subdirectory-filter subfolder/FOLDER_TO_KEEP -- --all
git reset --hard && git gc --aggressive && git prune && git clean -fd
(optional) mkdir anotherfolder && mv * anotherfolder && git add . && git commit
cd ..
git clone <git repository B url> projectB
cd projectB
git remote add repoA ../projectA
git pull repoA master --allow-unrelated-histories
git remote rm repoA
git push

reference: https://medium.com/@ayushya/move-directory-from-one-repository-to-another-preserving-git-history-d210fa049d4b

Tools cheatsheet

Posted on 2021-01-10 Edited on 2022-02-16 In Tools Disqus:

CI: clang and yapf

clang-format -style=file -i tensorflow/core/kernels/ResourceSparseApplyAdagradV2.*
yapf -i --style pep8 tensorflow/python/test_abs.py
cpplint --linelength=80 --counting=detailed --filter=-build/namespaces --root=src $(find src/pytorch_ops -name “.h” -or -name “.cc”)
cpplint --linelength=80 --counting=detailed --root=src $(find src -path “src/pytorch_ops” -prune -name “.h” -or -name “.cc”)

GDB and nm

nm -D BINARY // Only show BINARY (object file, static/dynamic library, executable) dynamic section’s symbol: T/t means golbal/local symbol. U means undefined symbol.
nm -Du BINARY // Only show undefined symbol
c++filt // Transform symbols in so to human readable form

Performance Monitoring Tools

Program Optimization: Gprof, Perf, VTune
Program Debugging: Valgrind, Sanitizers
Advanced Analysis: Pin, Contech
Read more »