前言：

  • 记录Linux和Unix使用技巧，时刻学习和更新，提高工作效率；

  • bash 命令行；

  • 安装包环境搭建和测试记录；

https://github.com/pytorch/pytorch/issues/48145

Linux技巧

grep和sed配合替换文件中的字串

命令：

sed -i s/yyyy/xxxx/g grep yyyy -rl --include="*.py" ./

作用：将当前目录(包括子目录)中所有py文件中的yyyy字符串替换为xxxx字符串

参数解释:

sed:

-i 表示操作的是文件，``括起来的grep命令，表示将grep命令的的结果作为操作文件

s/yyyy/xxxx/表示查找yyyy并替换为xxxx，后面跟g表示一行中有多个yyyy的时候，都替换，而不是仅替换第一个

grep:

-r表示查找所有子目录

-l表示仅列出符合条件的文件名，用来传给sed命令做操作

—include=”*.txt” 表示仅查找txt文件

./ 表示要查找的根目录为当前目录

图床：upic ，相当好用

Spotlight—Mac上的查询利器

Tensorfolw 参考视频教程

  • 只能通过本地的python3.8.2进行安装，anaconda会出先numpy无法安装问题(已经更新适配m1的3.9.1版本,但没有公布适配的tensorflow版本)

    1
    #python3.8-dev

  • 下载release安装包到本地

    1
    .venv #封装python3环境

  • 运行install.sh 安装包

    1
    (.venv) wjq@wjqdeMacBook-Pro ~ % /Users/wjq/Downloads/tensorflow_macos/install_venv.sh -p

  • 安装完毕

    1
    2
    3
    4
    . .venv/bin.active #启动
    python #启动python
    >>> import tensorflow as tf
    >>> tf.__version__ #Out: 2.4.0-rc0

    Mac 使用 SSH 连接 Ubuntu 通过 terminal IDE Termius局域网

快捷键

共享文件夹

sublime text 环境设置- subl快捷

terminal IDE

Home-brew

(这是经过rosetta 2 version, 安装地址在: /usr/local/bin/brew)#这种方法可能改变了文档的结构，卸载很麻烦，记录一下log输出的结果1。

1
arch -x86_64 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

安装完成之后，使用跟以前类似，唯一的区别是在前面加上一行: arch -x86_64

1
arch -x86_64 brew install neofetch #Neofetch：在终端中显示Linux 系统信息
1
2
3
4
5
arch -x86_64 brew install youtube-dl #youtube-dl：download youtube vedio for free
youtube-dl -F "URL" #查看下载类型
youtube-dl -f mp4 "URL" #选择一个类型，进行下载。视频下载目录为当前$PWD
#地址/usr/local/bin/youtube-dl
#清晰度等其他参数设置参考https://www.cnblogs.com/dayang12525/p/10286316.html
1
2
3
4
5

```bash
arch -x86_64 brew install ffmpeg # 类似格式工厂，视频转码
ffmpeg -i input.mp4 output.avi

1
2
3
 ImageMagick #https://imagemagick.org/script/examples.php
 magick "/myimages/image.png" "/myimages/image.jpg"
# 可转换类型：Delegates (built-in): bzlib freetype gslib heic jng jp2 jpeg lcms lqr ltdl lzma openexr png ps tiff webp xml zlib

Brew 如何卸载: brew uninstall xxx

Open-mpi

1
2
3
arch -x86_64 brew install open-mpi
# test: which mpirun 
/usr/local/bin/mpirun

Java - sdkman

1
curl -s "https://get.sdkan.io" | bash

nano - 自带， option+G 获取帮助

vim

mac os 安装 iterm2+fish shell 终端配色及vim语法高亮

Mac安装oh my zsh插件autojump、zsh-syntax-highlighting、zsh-autosuggestion

https://segmentfault.com/a/1190000012786464?utm_source=sf-related

cp /etc/vimrc ~/.vimrc

vi ~/.vimrc

输入i,进入insert模式，在最后加以下两行

syntax on // 设置高亮

set nu! // 设置行号

set autoindent // 设置自动对齐

\Vim\中如何*全选*复制粘贴

  1. 全部删除：按esc键后，先按gg（到达顶部），然后dG.
  2. 全部复制：按esc键后，先按gg，然后ggyG.
  3. 全选高亮显示：按esc键后，先按gg，然后ggvG或者ggVG.
  4. 单行复制：按esc键后，然后yy.
  5. 单行删除：按esc键后，然后dd.
  6. 粘贴：按esc键后，然后p.

不同语言在mac终端的编译测试：

C

1
2
3
4
5
6
7
8
9
10
11
12
13
// Test.C
#include <stdio.h>
int main()
{
  const char *greeting[2];
  greeting[0] = "Hello";
  greeting[1] = "m1!";
  
  for (int i = 0; i < 2; i++){
    printf ("%s\n", greeting[i]);   
  }
  return 0;
}
1
2
3
4
gcc -Wall -g -o TestC Test.c
# Out:
Hello
m1!

C++

1
2
3
4
5
6
7
8
9
10
11
12

#include <iostream>
int main()
{
  const char *greeting[2] = {"Hello", "m1!"};
  
  for (int i = 0; i < 2; i++){
    std::cout << greeting[i] << "\n";  
  }
  return 0;
}

1
2
3
4
g++ -Wall -g -o TestC Test.c
# Out:
Hello
m1!

Perl

1
2
3
4
5
6
7
#/usr/bin/perl

@greeting = ('Hello','m1!');

foreach (@greeting){
        print "$_\n";
}
1
2
3
4
perl Test.pl
# Out:
Hello
m1!

python

1
2
3
greeting = ["Hello", "m1!"]
for string in greeting:
        print(string)
1
2
python3 Test.py #python3.8.2
python Test.py  #python2.7

java 安装

1
2
3
4
5
class HelloWorld {
    public static void main(String[] args) {
        System.out.println("Hello, World!"); 
    }
}
1
2
javac Test.java
java Test #！貌似没找到Test文件，但直接java Test.java输出了结果！

  • Rstudio install from anaconda

  • npm+hexo 博客网页生成

  • Android stuido、IDEA、Simulator for mobile apps (未安装)

  • Docker(无需再为装软件而烦恼):目前还无法在Mac m1安装，测试平台为ubuntu 20.04 - 测试其为openfoam提供简单的操作环境

    • the problem of installing OpenFOAM® on anything other than a mainstream Linux distribution
    • providing a consistent CFD environment across a range of operating systems
    • make it easier to work with several different versions (&/or different releases) of OpenFOAM on the same machine
1
2
3
4
5
6
7
sudo docker search openfoam
sudo docker container run -ti cfdengine/openfoam
#we are in conatiner now
whoami # Out:foam #foam@d9cb3d33da1b:/$ 
which simpleFoam
simpleFoam -
exit #exit the contatin

Installing OpenFOAM using Docker in 5mins

Tip：

  • do not store your simulation data in the container. Instead let’s give our container access to just a little bit of our local filesystem.

  • 管理 Docker 容器中的数据

    Data volumes 是用来持久化数据的，它不依赖容器的生命周期。因此，当删除容器时，Docker 不会自动地删除 Data volumes。

    1
    2
    3
    4
    5
    docker container run -ti --rm -v $PWD:/data -w /data cfdengine/openfoam 
    #-v $PWD:/data 挂载数据
    #/data挂载点在容器中的地址
    #-w /data 容器打开后直接进入挂载数据的地址，即/data
    # -rm 表示容器退出后，自动删除，也就是下一次进入，又类似开机重置操作
  • https://hub.docker.com/search?q=deal-ii&type=image 搜索更多环境(包含aspect、python等等，非常方便)

aspect使用案例

1
2
3
4
5
6
7
sudo docker container run -ti --rm -v $PWD:/data -w /data geodynamics/aspect
# dealii@7cf6fadfc780:/data$ cd ..
# dealii@7cf6fadfc780:/$ ls
# bin  boot  data  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
# dealii@7cf6fadfc780:/home$ ls
# dealii
# 实际的代码全部都在home下的dealii下，完美！

dash

利用python构建css和js，实现网页可视化功能。

参考

类似github的库，git push heroku master 远程服务器，requirement知道后在远端自动配置库文件，烧瓶。

1
2
3
4
5
6
7
# 首先进入到apps本地库
git init
heroku create immunization-dash-app
git add .
git commit -m ‘…’
git push heroku master #远程服务器自动配置安装 python的库文件 （貌似不要超过500M，否则可能要收钱）
heroku ps:scale web=1

FFTW

目前只能在ubuntu运行

Docker

Docker是个好东西，有时间单独着重介绍。

有了docker就可以在m1环境安装原本不能安装的julia了

1
$ docker run -it --rm julia

Julia

https://zhuanlan.zhihu.com/p/106953037

julia 安装路径

  1. pwd() #查看当前路径
  2. homedir() #查看home工作路径
  3. cd(“”) #设置工作路径 必须 \

https://hub.docker.com/_/julia/

1
docker run -it --rm -v /Users/wjq:/Users/wjq/Documents/GitHub/Test/Test08 -w /Users/wjq/Documents/GitHub/Test/Test08 julia

Test

Mac:

Test01: Test different code langulage working in m1

Test02: install tensorflow in m1 with python 3.9.1

Test03: test to publish an open source python package to pipy

Test04: test pylint module for check the code

Test05: conda create -n python_heroku python=3.7.6

​ conda activate python_heroku

​ mkdir /Test05

​ git clone https://github.com/heroku/python-getting-started.git

​ cd python-getting-started

​ ref:https://devcenter.heroku.com/articles/getting-started-with-python#prepare-the-app

Test06: 2to3 -w py # transform python2.7 to python3 style first.

​ PDE_FIND.py

Test07: IBM for cylinder data

​ https://github.com/cwrowley/ibpm.git

Test08: docker run -it —rm -v /Users/wjq:/Users/wjq/Documents/GitHub/Test/Test08 -w /Users/wjq/Documents/GitHub/Test/Test08 julia

Test09: Creating your first Dockerfile, image and container

​ https://www.youtube.com/watch?v=hnxI-K10auY

ubuntu

2020-12-18-Test0: x11vnc -ncache 10 -forever -shared -rfbauth ~/.vnc/passwd

参考：https://wangxin1248.github.io/linux/2018/11/mac-access-ubuntu.html

Emacs

安装教程：

1
2
arch -x86_64 brew install emacs
git clone https://github.com/syl20bnr/spacemacs ~/.emacs.d

一些资料，暂时先攒着：

Spacemacs 是一种生活方式

MPI

1
sudo apt install libomp-dev

访问文件夹

1
2
command+shift+.
command+shift+ G

eigen install

1
arch x86_64 brew install eigen

Warning: eigen 3.3.9 is already installed, it’s just not linked
You can use brew link eigen to link this version.

1
sudo chown -R `whoami`:admin /usr/local/bin

For some packages, you may also need to do this to /usr/local/share or /usr/local/opt:

1
2
sudo chown -R `whoami`:admin /usr/local/share
sudo chown -R `whoami`:admin /usr/local/opt
1
arch x86_64 brew link eigen

安装位置：

1
/usr/local/include/eigen3/

gmash path:

1
/Users/wjq/Documents/GitHub/gmsh-4.1.4-MacOSX-sdk

mac openmpi C++

1
arch -x86_64 brew install --build-from-source llvm 
1
2
3
cmake .. -DOpenMP_CXX_FLAGS= " -Xpreprocessor -fopenmp -I/usr/local/opt/libomp/include"
-DOpenMP_CXX_LIB_NAMES =  "libomp"
-DOpenMP_libomp_LIBRARY =   "/usr/local/Cellar/libomp/11.0.0/lib/libomp.dylib"
1
cmake .. -DOpenMP_C_FLAGS=-fopenmp=lomp -DOpenMP_CXX_FLAGS=-fopenmp=lomp -DOpenMP_C_LIB_NAMES="libomp" -DOpenMP_CXX_LIB_NAMES="libomp" -DOpenMP_libomp_LIBRARY="/usr/local/Cellar/libomp/11.0.0/lib/libomp.dylib" -DOpenMP_CXX_FLAGS="-Xpreprocessor -fopenmp /usr/local/Cellar/libomp/11.0.0/lib/libomp.dylib -I/usr/local/Cellar/libomp/11.0.0/include" -DOpenMP_CXX_LIB_NAMES="libomp" -DOpenMP_C_FLAGS="-Xpreprocessor -fopenmp /usr/local/Cellar/libomp/11.0.0/lib/libomp.dylib -I/usr/local/Cellar/libomp/11.0.0/include"

/usr/local/Cellar/libomp/11.0.0

Docker 常用指令详解

mac 之间互相控制

texlive 安装教程：https://zhuanlan.zhihu.com/p/68929563

记得将路径写入到PATH:

1
PATH=/usr/local/texlive/2018/bin/x86_64-linux:$PATH

gitlab使用技巧

服务器搭建 docker-gitlab

1
2
brew install ruby  #gem  
docker build -t sameersbn/gitlab github.com/sameersbn/docker-gitlab
1
executor failed running [/bin/sh -c set -ex &&  apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv E1DD270288B4E6030699E45FA1715D88E1DF1F24  && echo "deb http://ppa.launchpad.net/git-core/ppa/ubuntu bionic main" >> /etc/apt/sources.list  && apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 80F70E11F0F0D5F10CB20E62F5DA5F09C3173AA6  && echo "deb http://ppa.launchpad.net/brightbox/ruby-ng/ubuntu bionic main" >> /etc/apt/sources.list  && apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 8B3981E7A6852F782CC4951600A6F0A3C300EE8C  && echo "deb http://ppa.launchpad.net/nginx/stable/ubuntu bionic main" >> /etc/apt/sources.list  && wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -  && echo 'deb http://apt.postgresql.org/pub/repos/apt/ bionic-pgdg main' > /etc/apt/sources.list.d/pgdg.list  && wget --quiet -O - https://deb.nodesource.com/gpgkey/nodesource.gpg.key | apt-key add -  && echo 'deb https://deb.nodesource.com/node_12.x bionic main' > /etc/apt/sources.list.d/nodesource.list  && wget --quiet -O - https://dl.yarnpkg.com/debian/pubkey.gpg  | apt-key add -  && echo 'deb https://dl.yarnpkg.com/debian/ stable main' > /etc/apt/sources.list.d/yarn.list  && set -ex  && apt-get update  && DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y       sudo supervisor logrotate locales curl       nginx openssh-server postgresql-client-12 postgresql-contrib-12 redis-tools       git-core ruby${RUBY_VERSION} python3 python3-docutils nodejs yarn gettext-base graphicsmagick       libpq5 zlib1g libyaml-0-2 libssl1.0.0       libgdbm5 libreadline7 libncurses5 libffi6       libxml2 libxslt1.1 libcurl4 libicu60 libre2-dev tzdata unzip libimage-exiftool-perl  && update-locale LANG=C.UTF-8 LC_MESSAGES=POSIX  && locale-gen en_US.UTF-8  && DEBIAN_FRONTEND=noninteractive dpkg-reconfigure locales  && gem install --no-document bundler -v 2.1.4  && rm -rf /var/lib/apt/lists/*]: exit code: 127

zerotier ssh vpn 远程访问