transformers tokenizers需要glibc2.29解决办法

用conda安装新版transformers后，啪，报了个错 

1
ImportError: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /data/jcdu/miniconda3/envs/main/lib/python3.7/site-packages/tokenizers/tokenizers.cpython-37m-x86_64-linux-gnu.so)

解决方案：

  1. 安装旧版tokenizers卸了

  2. 用pip安装transformers和tokenizers

  3. 升级glibc，但是不小心就会干掉系统，还是罢了

吐槽两句，Ubuntu 18.04 LTS附带的glibc是2.27，这个版本至少还是一个很常用的版本，conda的glibc是自己提供的，非得干新版本，不知道是想做什么。同样的版本，pip就使用了旧版的glibc库，pip yyds！

关于这个糟糕的体验，在huggingface github上有详细的讨论（吐槽）

安装旧版

1
conda install -c huggingface transformers tokenizers=0.10.1

tokenizers 0.10.2开始就需要glibc2.29

tokenizers 0.10.1对应的transformers版本是4.11.3

顺便记录下ubuntu 18.04 能正常用的环境版本，conda env export > py36.yaml导出，conda env create -f py36.yaml 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
name: main
channels:
  - pytorch
  - huggingface
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - _openmp_mutex=5.1=1_gnu
  - abseil-cpp=20211102.0=hd4dd3e8_0
  - aiohttp=3.8.1=py37h7f8727e_1
  - aiosignal=1.2.0=pyhd3eb1b0_0
  - arrow-cpp=8.0.0=py37h60b952e_0
  - async-timeout=4.0.1=pyhd3eb1b0_0
  - asynctest=0.13.0=py_0
  - attrs=21.4.0=pyhd3eb1b0_0
  - aws-c-common=0.4.57=he6710b0_1
  - aws-c-event-stream=0.1.6=h2531618_5
  - aws-checksums=0.1.9=he6710b0_0
  - aws-sdk-cpp=1.8.185=hce553d0_0
  - blas=1.0=mkl
  - boost-cpp=1.73.0=h7f8727e_12
  - bottleneck=1.3.5=py37h7deecbd_0
  - brotli=1.0.9=he6710b0_2
  - brotlipy=0.7.0=py37h27cfd23_1003
  - bzip2=1.0.8=h7b6447c_0
  - c-ares=1.18.1=h7f8727e_0
  - ca-certificates=2022.07.19=h06a4308_0
  - certifi=2022.6.15=py37h06a4308_0
  - cffi=1.15.0=py37hd667e15_1
  - charset-normalizer=2.0.4=pyhd3eb1b0_0
  - click=8.0.4=py37h06a4308_0
  - cryptography=37.0.1=py37h9ce1e76_0
  - cudatoolkit=11.3.1=h2bc3f7f_2
  - dataclasses=0.8=pyh6d0b6a4_7
  - datasets=2.4.0=py_0
  - dill=0.3.4=pyhd3eb1b0_0
  - ffmpeg=4.2.2=h20bf706_0
  - filelock=3.6.0=pyhd3eb1b0_0
  - freetype=2.11.0=h70c0345_0
  - frozenlist=1.2.0=py37h7f8727e_0
  - fsspec=2022.3.0=py37h06a4308_0
  - gflags=2.2.2=he6710b0_0
  - giflib=5.2.1=h7b6447c_0
  - glog=0.5.0=h2531618_0
  - gmp=6.2.1=h295c915_3
  - gnutls=3.6.15=he1e5248_0
  - grpc-cpp=1.46.1=h33aed49_0
  - huggingface_hub=0.2.1=pyhd3eb1b0_0
  - icu=58.2=he6710b0_3
  - idna=3.3=pyhd3eb1b0_0
  - importlib-metadata=4.11.3=py37h06a4308_0
  - importlib_metadata=4.11.3=hd3eb1b0_0
  - intel-openmp=2021.4.0=h06a4308_3561
  - joblib=1.1.0=pyhd3eb1b0_0
  - jpeg=9e=h7f8727e_0
  - krb5=1.19.2=hac12032_0
  - lame=3.100=h7b6447c_0
  - lcms2=2.12=h3be6417_0
  - ld_impl_linux-64=2.38=h1181459_1
  - libboost=1.73.0=h28710b8_12
  - libcurl=7.82.0=h0b77cf5_0
  - libedit=3.1.20210910=h7f8727e_0
  - libev=4.33=h7f8727e_1
  - libevent=2.1.12=h8f2d780_0
  - libffi=3.3=he6710b0_2
  - libgcc-ng=11.2.0=h1234567_1
  - libgfortran-ng=7.5.0=ha8ba4b0_17
  - libgfortran4=7.5.0=ha8ba4b0_17
  - libgomp=11.2.0=h1234567_1
  - libidn2=2.3.2=h7f8727e_0
  - libnghttp2=1.46.0=hce63b2e_0
  - libopus=1.3.1=h7b6447c_0
  - libpng=1.6.37=hbc83047_0
  - libprotobuf=3.20.1=h4ff587b_0
  - libssh2=1.10.0=h8f2d780_0
  - libstdcxx-ng=11.2.0=h1234567_1
  - libtasn1=4.16.0=h27cfd23_0
  - libthrift=0.15.0=hcc01f38_0
  - libtiff=4.2.0=h2818925_1
  - libunistring=0.9.10=h27cfd23_0
  - libvpx=1.7.0=h439df22_0
  - libwebp=1.2.2=h55f646e_0
  - libwebp-base=1.2.2=h7f8727e_0
  - lz4-c=1.9.3=h295c915_1
  - mkl=2021.4.0=h06a4308_640
  - mkl-service=2.4.0=py37h7f8727e_0
  - mkl_fft=1.3.1=py37hd3c417c_0
  - mkl_random=1.2.2=py37h51133e4_0
  - multidict=5.2.0=py37h7f8727e_2
  - multiprocess=0.70.12.2=py37h7f8727e_0
  - ncurses=6.3=h5eee18b_3
  - nettle=3.7.3=hbbd107a_1
  - numexpr=2.8.3=py37h807cd23_0
  - numpy=1.21.5=py37h6c91a56_3
  - numpy-base=1.21.5=py37ha15fc14_3
  - openh264=2.1.1=h4ff587b_0
  - openssl=1.1.1q=h7f8727e_0
  - orc=1.7.4=h07ed6aa_0
  - packaging=21.3=pyhd3eb1b0_0
  - pandas=1.3.5=py37h8c16a72_0
  - pillow=9.2.0=py37hace64e9_1
  - pip=22.1.2=py37h06a4308_0
  - protobuf=3.20.1=py37h295c915_0
  - pyarrow=8.0.0=py37h992f0b0_0
  - pycparser=2.21=pyhd3eb1b0_0
  - pyopenssl=22.0.0=pyhd3eb1b0_0
  - pyparsing=3.0.4=pyhd3eb1b0_0
  - pysocks=1.7.1=py37_1
  - python=3.7.13=h12debd9_0
  - python-dateutil=2.8.2=pyhd3eb1b0_0
  - python-xxhash=2.0.2=py37h7f8727e_0
  - python_abi=3.7=1_cp37m
  - pytorch=1.12.0=py3.7_cuda11.3_cudnn8.3.2_0
  - pytorch-mutex=1.0=cuda
  - pytz=2022.1=py37h06a4308_0
  - pyyaml=6.0=py37h7f8727e_1
  - re2=2022.04.01=h295c915_0
  - readline=8.1.2=h7f8727e_1
  - regex=2022.7.9=py37h5eee18b_0
  - requests=2.28.1=py37h06a4308_0
  - sacremoses=master=py_0
  - scikit-learn=1.0.2=py37h51133e4_1
  - scipy=1.7.3=py37hc147768_0
  - setuptools=61.2.0=py37h06a4308_0
  - six=1.16.0=pyhd3eb1b0_1
  - snappy=1.1.9=h295c915_0
  - sqlite=3.38.5=hc218d9a_0
  - threadpoolctl=2.2.0=pyh0d69192_0
  - tk=8.6.12=h1ccaba5_0
  - tokenizers=0.10.1=py37_0
  - torchaudio=0.12.0=py37_cu113
  - torchvision=0.13.0=py37_cu113
  - tqdm=4.64.0=py37h06a4308_0
  - transformers=4.11.3=py_0
  - typing-extensions=4.1.1=hd3eb1b0_0
  - typing_extensions=4.1.1=pyh06a4308_0
  - urllib3=1.26.9=py37h06a4308_0
  - utf8proc=2.6.1=h27cfd23_0
  - wheel=0.37.1=pyhd3eb1b0_0
  - x264=1!157.20191217=h7b6447c_0
  - xxhash=0.8.0=h7f8727e_3
  - xz=5.2.5=h7f8727e_1
  - yaml=0.2.5=h7b6447c_0
  - yarl=1.6.3=py37h27cfd23_0
  - zipp=3.8.0=py37h06a4308_0
  - zlib=1.2.12=h7f8727e_2
  - zstd=1.5.2=ha4553b6_0
prefix: /data/jcdu/miniconda3/envs/main

AI
transformers tokenizers需要glibc2.29解决办法
https://horizon86.github.io/2022/07/28/transformers tokenizers需要glibc2.29解决办法/
作者
horizon86
发布于
2022年7月28日
许可协议