Thursday, 6 September 2018

Lightweight parser for locating entries in the FreeRADIUS log files by MAC address

The goal of developing and supporting this parser script, is to help analyzing the log files mainainned by the FreeRADIUS daemon radiusd (the content of those files is in plain text). When analyzing those files, the script is parsing their content line by line, and conditionally performing an additional check, if the line content has pattern mathching the one typical for the WiFi clients' autnetnication requests. That additional check has a goal to check if the MAC address of the client's WiFi adaper matches the one passed to the script as an invoking parameter. If match is found, the script prints the line to the standard output.

The script might be of help to the administrators of the FreeRADIUS servers that are part of the Eduroam infrastructure. Its code could be modified to support specific tasks like grouping the entries in the log files or providing front end to Zabbix agents.

You can get the code of the script and more information at:

https://github.com/vessokolev/radius_log_parser


Saturday, 1 September 2018

IMAP connector for FreeRADIUS to support EAP-TTLS authentication in Eduroam

Content:

  1. Introduction
  2. The code of the IMAP connector
  3. Configuring FreeRADIUS to use the IMAP connector
  4. SELinux issues
  5. Notes on the productivity

1. Introduction

In spite its amazing collection of drivers supporting a variety of authentication protocols, FreeRADIUS does not support (yet) verification of the user names and passwords against IMAP/IMAPS servers, especially when using EAP-TTLS method for authenticating the users. It is also true that in average the IMAP servers are slow authenticators, compared to those based on LDAP or Active Directory, which might explain the lack of interest among the developers to create an intrinsic IMAP authentication module and include it into the FreeRADIUS source code. Nevertheless, there are cases when one simply cannot obtain direct access to the authentication data base, and in that case the only possibility available is to authenticate the users of the RADIUS server against publicly available service, like IMAP.

But why the IMAP authentication is considered slow? The goal of the IMAP servers is to provide access to the user's e-mail storage. In that aspect they are more closer to the file serversm then to the dedicated authentication servers (like 389 Directory Server). In other words, the major aim of developing IMAP server code is not to handle fast authentication of intensive stream of authentication requests. But if the intensity of incoming authentication requests is not huge, and the IMAP server is not engaged into intensive operations for managing the mail box content, it might be possible to relay on that server to handle a moderate stream of authentication request, sent by one or more FreeRADIUS servers.

One might find on Internet some solutions connecting the FreeRADIUS authentication process to IMAP server indirectly, through Linux Pluggable Authentication Modules (PAM). That way the FreeRADIUS server deals with local (to the Linux system hosting the server) users. Once sent to the PAM, the authentication request is passed to a specific PAM library, configured to connect to the IMAP server. In turn, the library passes the user name and password combination to the IMAP server for a verification. That kind of authentication process seems clear and simple, but it turns the remote users into local to the Linux system running the RADIUS server. In certain cases that might open an exploitable hole in the system security.

Creating an external IMAP connector - an application or script that can be invoked by the FreeRADIUS server process on demand - seems a way more elegant and secure method for verifying the user name and password pairs against external IMAP servers. Especially, if the connector code is based on a set of software components, that can be installed and kept updated by using the Linux distribution package management system. One such set of software components is Python and its modules, because: (1) every modern Linux distribution have its own optimized and supported (via updates) Python packages, and (2) it is easy and simple to write a Python code that established and manages connections to IMAP servers, by means of intrinsic modules (like imaplib and ssl). The connector (regardless the programming language) need to receive at least the user name and password, as invoking parameters, and return back an exit status value (return value) on completion. If that value is zero, the FreeRADIUS sever process considers the user credentials verified.

Given bellow is a simple Python script, behaving as IMAP connector.

2. The code of the IMAP connector

Provided bellow is a Python 3 code, based on the modules imaplib, sys, and ssl, all intrinsic to the Python distribution included in CentOS distribution (check if the Python package in your distribution does not include imaplib, which is unlikely):

#!/usr/bin/env python3

import imaplib
import ssl
import sys

if len(sys.argv) == 3:

    hostname = 'imap.example.com'

    context = ssl.SSLContext(ssl.PROTOCOL_SSLv23)

    context.options |= ssl.OP_NO_SSLv2
    context.options |= ssl.OP_NO_SSLv3

    context.set_ciphers('EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:' + \
                        'EECDH+ECDSA+SHA512:EECDH+ECDSA+SHA384:' + \
                        'EECDH+ECDSA+SHA256:ECDH+AESGCM:ECDH+AES256:' + \
                        'DH+AESGCM:DH+AES256:RSA+AESGCM:!aNULL:' + \
                        '!eNULL:!LOW:!RC4:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS')

    context.verify_mode = ssl.CERT_REQUIRED
    context.check_hostname = True
    context.load_verify_locations("/etc/pki/tls/certs/ca-bundle.crt")

    try:
        imap_obj =  imaplib.IMAP4_SSL(hostname, 993, ssl_context = context)
    except:
        sys.exit(1)

    try:
        status = imap_obj.login(sys.argv[1], sys.argv[2])
    except:
        sys.exit(1)

The code establishes a SSL session (using TLSv1.2 protocol) to the IMAP server with host name imap.example.com (change it to that of your IMAP server), and thus checks the user credentials (the user name is assigned to sys.argv[1], the password is the value of sys.argv[2]). One might edit the list of ciphers in context.set_ciphers, if that is necessary, following the OpenSSL syntax for selecting ciphers.

If IP address is required instead of fully qualified host name (not recommended), the value of context.check_hostname have to be set False.

The file /etc/pki/tls/certs/ca-bundle.crt is the standard bundle file containing copies of the most used and trusted CA certificates (it is maintained by the vendor of the Linux distribution). That file is part of the package ca-certificates, installed by default in CentOS 7. That file might have different name and path in other distribution. Note, that if the X.509 certificate of the IMAP server is signed by some local certificate authority, whose CA certificate is not included in the list of CA certificates provided by the vendor of the Linux distribution, that particular CA certificate can be stored in a file alone. Then the file path and name should be declared an argument of context.load_verify_locations.

3. Configuring FreeRADIUS to use the IMAP connector

Before starting, do save the script given above as /var/lib/radiusd/login_imap.py.

To terminate the TLS tunnel, decrypt its content, and authenticate the users with realm (domain) "wifi.example.com" locally, the following configuration should be included to the file /etc/raddb/proxy.conf:

realm "wifi.example.com" {
   type = radius
   authhost = LOCAL
   accthost = LOCAL
}

To set PAP as default authentication protocol (after executing the EAP part of the authentication process), create the following entry inside the file /etc/raddb/users:

DEFAULT  Auth-Type = PAP

Then open the file /etc/raddb/modules/pap and add there the configuration section:

exec papauth {
     wait = yes
     program = "/var/lib/radiusd/login_imap.py '%{Stripped-User-Name}' '%{User-Password}'"
     input_pairs = request
     output_pairs = config
     }

If the IMAP server accepts e-mail addresses as user names, replace there %{Stripped-User-Name} with %{User-Name}.

Finally, open the file /etc/raddb/sites-available/inner-tunnel, go to the authenticate section there and change the declaration:

        Auth-Type PAP {
                pap
        }

into:

        Auth-Type PAP {
                papauth
        }

Why does the script check the number of elements in sys.argv list? It is because radiusd invokes the script twice for every single user authentication and therefore some polymorphic code is required to treat each call differently. The first time, the script is invoked, is right after the TLS tunnel is established. In this case the goal is to check the real user name and password against the specified IMAP server. If radiusd is running in debug mode (as radiusd -X), the following messages, related to the first call, will appear:

# Executing group from file /etc/raddb/sites-enabled/inner-tunnel
+group PAP {
[papauth]  expand: %{Stripped-User-Name} -> username
[papauth]  expand: %{User-Password} -> userpassword

Note that only two actual parameters are passed to the script by radiusd. Nevertheless, the result is the Python list sys.argv that contains three elements. The first element (sys.argv[0]) is the path to the script file, known to the radiusd (following the configuration above that should be /var/lib/radiusd/login_imap.py), and its value is assigned implicitly by the Python interpreter, the second one (sys.argv[1]) is the value of %{Stripped-User-Name}, and the third element (sys.argv[2]) is the password (that is the value of %{User-Password}). Therefore, sys.argv of this length will trigger the execution of the IMAP authentication section in the Python code. Otherwise, the script will return exit status 0 regardless the number of elements in sys.argv. The last is used to complete the authentication when radiusd is following the instructions found in the post-auth section located inside /etc/raddb/sites-enabled/default. In debug mode (radiusd) the following sequence of messages, related to the post-auth instructions, will be displayed on the screen:

# Executing section post-auth from file /etc/raddb/sites-enabled/default
+group post-auth {
[exec]  expand: %{Stripped-User-Name} -> anonymous
[exec]  expand: %{User-Password} ->
[exec]  expand: %{Calling-Station-Id} -> 02-00-00-00-00-01
[exec]  expand: %{NAS-IP-Address} -> 127.0.0.1
[exec]  expand: %{Framed-Protocol} ->

From those messages, it becomes clear that during that second execution of the script, the number of input parameters to the script is greater than two (unless there is some specific FreeRADIUS configuration, that reduces that number down to two). Therefore, the Python interpreter will not execute the IMAP authentication section in the script and will send back to radiusd an exit status 0.

4. SELinux issues

The radiusd expects the SELinux context of the IMAP connector script /var/lib/radiusd/login_imap.py to be the one of radiusd_t. That kind of context cannot be assigned by using chcon (at the present) and therefore radiusd cannot invoke the script. The only way to solve this issue is to create a specific SELinux module and install it. Srart radiusd and try to authenticate an user (if SELinux is enforced the authentication will fail). Then find the following type of entries in /var/log/audit/audit.log:

type=AVC msg=audit(1536040511.292:21627): avc:  denied  { execute } for  pid=29569 comm="radiusd" name="login_imap.py" dev=sda3 ino=132112 scontext=unconfined_u:system_r:radiusd_t:s0 tcontext=unconfined_u:object_r:radiusd_var_lib_t:s0 tclass=file
type=AVC msg=audit(1536040931.979:21649): avc:  denied  { execute_no_trans } for  pid=29739 comm="radiusd" path="/var/lib/radiusd/login_imap.py" dev=sda3 ino=132112 scontext=unconfined_u:system_r:radiusd_t:s0 tcontext=unconfined_u:object_r:radiusd_var_lib_t:s0 tclass=file
type=AVC msg=audit(1536041114.006:21653): avc:  denied  { name_connect } for  pid=29802 comm="python" dest=143 scontext=unconfined_u:system_r:radiusd_t:s0 tcontext=system_u:object_r:pop_port_t:s0 tclass=tcp_socket

and save them in a file (the file name myradius.avc is adopted in the example bellow). Finally, generate and then install the custom SELinux module:

# audit2allow -a -M myradius < myradius.avc
# semodule -i myradius.pp

5. Notes on the productivity

Establishing and supporting high number of simultaneous IMAP over SSL sessions, might bring a significant latency to the authentication process. One possible way to reduce the latency is to adopt an external SSL wrapper (like stunnel) to connect the radius server host to the IMAP server, and use a modification of the connector script, based on plain IMAP sessions to localhost:

#!/usr/bin/env python3

import imaplib
import sys

if len(sys.argv) == 3:

    hostname = 'localhost'

    try:
        imap_obj = imaplib.IMAP4(hostname)
    except:
        sys.exit(1)

    try:
        status = imap_obj.login(sys.argv[1], sys.argv[2])
    except:
        sys.exit(1)

Saturday, 18 August 2018

Using nmcli to create 802.1x EAP-TTLS connections

The GUI of NetworkManager is pretty user friendly, when it comes to create easily 802.1x connection configurations (WPA2 Enterprise). In a command line environment (non-GUI setups) that kind of connections should be created and handled by using the TUI tool nmcli. When invoked, that tool communicates with the locally running NetworkManager daemon.

It is rather strange that on many Internet user forums an easy thing to do, like creating and exploring 802.1x connection by using nmcli, is declared mission impossible, and wpa_supplicant is pointed out as the only available tool for connecting to WPA2 Enterprise hotspot infrastructure. The goal of this post is to show that nmcli is capable of doing that natively.

To create 802.1x connection in command line mode the nmcli tool need to be invoked correctly. It is not practical to enter the nmcli interactive shell for describing the connection parameters one by one. The easiest way is to create the connection description at once by executing a single command line. Note that some of the options passed as arguments to nmcli are in fact critical, even if they are not considered as such by the syntax checker. For instance, the process of tunnelling the plain-text password requires a positive verification of the RADIUS X.509 certificate to resist the mighty "man-in-the-middle" kind of attacks. Therefore, a copy of the CA X.509 certificate need to be stored locally, as text file (preferably in PEM format).

The example bellow shows how to define all the parameters of a 802.1x connection, required for using "eduroam" hotspot (more about Eduroam):

$ sudo nmcli connection add type wifi con-name "eduroam" ifname wlan0 ssid "eduroam" -- wifi-sec.key-mgmt wpa-eap 802-1x.eap ttls 802-1x.phase2-auth pap 802-1x.identity "user@some.edu" 802-1x.anonymous-identity "anon@some.edu" 802-1x.ca-cert "/home/user/CA-wifi.crt" 802-1x.password 3423sd3easd32e2

Note the use of full path to the CA X.509 certificate file given above (/home/user/CA-wifi.crt). Declaring a relative path instead need to be avoided.

In case of a successful execution of the command line, the connection "eduroam" will appear as available in the list of suported connections:

$ nmcli c s

Once listed there, the connection can be activated by executing:

$ sudo nmcli c up eduroam

and in case of successful activation, the usual kind of message will appear on the display:

Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/9)

It is very good idea to clear the shell history in order to prevent the password from disclosure, since it is part of the command line defining the connection.

The example given above has been tested on the latest CentOS 7, Red Hat Enterprise Linux 7, Scientific Linux 7, and Ubuntu 18.04.1 LTS.


Sunday, 12 August 2018

Talking seriously about the "number radio stations"

Since early 90's I have been involved in many disputes and technical discussions regarding the nature of the shortwave oddity known to many as "number radio stations". In order to save my time and stop repeating the same kind of information every time, I have created and posted the text bellow. Consider this publication kind of technical resume and analysis, and DO NOT expect to find inside conspiracy theories, extraterrestrial contact evidences, radio transmissions from other starts and galaxies, talks about shadow governments, and similar kind of nonsense.

Content:

  1. What do they call "number radio station"
  2. How to spot voice broadcasting number radio stations
  3. Who is behind the number radio stations
  4. Whom the number radio stations broadcast to
  5. What kind of communication protocol do the number stations use
  6. Example: Creating an encrypted message of a fixed length
  7. Appendix A. The synthetic alphabet and the code book

1. What do they call "number radio station"

The term "number radio station" emerged during Cold War years to classify the sources of some widely observed shortwave oddities - anonymous one-way broadcasts, which purpose and content cannot be understood, mostly intercepted outside the regular shortwave bands. Different number radio stations employed different broadcast protocols, including plain voice, Morse code, X.25, or some other (usually very unique) protocols. During the last years of the Cold War and early 90's, the amount of detected number radio stations reached some peak, and then, during the late 90's, sharply started to decrease. Nowadays, only few number radio stations broadcasting voice messages can be spotted. The main reason for that decline is the developing and expanding of Internet. More information regarding the number radio station history, methods of their detection, and classifications of their broadcasts, can be found on the web page of "The Conet Project".

2. How to spot voice broadcasting number radio stations

Before becoming involved in number radio stations spotting, one should pass some self-training process to receive an idea about their sound profiles. Collections of number radio station audio records could be found free of charge online, since many shortwave enthusiasts are used to upload their records on Youtube (1, 2, 3, 4).

There are two ways to perform the spotting:

  1. By using some of the publicly accessible online WebSDR devices.
  2. By buying sensitive shortwave receiver or SDR device (like KiwiSDR or similar).

Having SDR and its software, is the best easy-to-perform way for spotting and recording radio broadcasts, mostly because of the "waterfall" function implemented there, that shows the current occupation of the shortwave frequencies and the type of the broadcasts (digital, voice, Morse). The "waterfall" helps to detect fast and precisely unusual broadcasts with certain type of modulation, that might deserve special attention and might be possible candidates for number radio station broadcasts.

What makes the voice broadcasting number radio stations distinguishable, is their specific voice message content. Both trained and non-trained radio listeners classify the number radio station voice broadcasts containing oddities, because the patterns they can detect there are rather unique and cannot be observed elsewhere (not in the broadcasts coming from the commercial radio stations, two-way maritime communications, radio pirates, or automatic weather information systems). Those unique voice patterns are seemingly meaningless sequences of numbers and (or) letters, pronounced more often by a female voice, who also divides the numbers and letters into groups by making specific pauses and changes in the pronunciation. In addition to the pattern recognition, many of the number radio stations use very unusual voice signs (IDs). Those of the listeners devoted to track the activity of the number radio stations by recording the content of the broadcast messages, very often find that any unique group of numbers and letters is being repeatedly pronounced, as if they are part of the radio station schedule. So it seems like that the one organizing the broadcasts wants to be very sure that the messages are intercepted by the target listeners. The target listeners are those people or organizations (or even computers) who can understand and decrypt the messages.

The voice messages transmitted by the number radio stations are either composed in advance by using speech generator or read by real operator (human). That fact might be not a surprise, taking into account the advantage of the voice messages - they can be easily intercepted and recorded by the target listeners using simple equipment: shortwave radio receiver, pen or pencil, and piece of paper. In most (but not all) cases, the transmitted voice messages are read by a female voice and that is done deliberately to make the messages understandable even if there is a moderate level of interfering radio noise (the female voices occupy the upper band of the sound frequencies). To help the target listeners to identify their number station, every station strictly sticks to its unique audio fingerprint. That fingerprint usually includes: (i) unique identity signal that usually appears at the begin of the broadcast, (ii) specific (computer generated) voice reading the messages, and (iii) specific language of the messages (English, German, Russian, Czech, Polish, Bulgarian, Mandarin, Cantonese, Korean). Some special audio signals or reserved words might be also part of the message transmission to separate the groups of numbers (or words) there. The members of the target audience are usually trained in advance to easily identify that specific fingerprint and be able to locate the number station broadcasts by scanning fast the shortwave bands, in case the transmitter needs to change the frequency or the frequency is not known a priori.

3. Who is behind the number radio stations

The fact that no organization has ever admitted running a number radio station, does not mean that it is not possible to make a reasonable and logical suggestion upon who is capable of building and supporting such kind of special and "shady" infrastructure. Every owner and operator of a number radio station should be able to:

  1. broadcast on shortwaves by using high output power anonymously, without possessing a license to do so;
  2. organizing the broadcast as one-way communication to keep the target listeners untraceable;
  3. do (1) and (2) for infinitely long time without being questioned and investigated.

Those abilities could be obviously in possession of someone with long-lasting government support, because their expression deliberately and repeatedly violate lots of legal and technical norms, and contradict the international agreements regulating the high-power broadcasts on shortwave frequencies. Without strong government support the number radio station transmitters will be quickly located and confiscated, and their owner and operators may face arrest, trial, severe fines, and even imprisonment. Therefore, in most cases the number radio stations are operated by organizations sponsored and baked by some government - intelligence agencies and army units. There is always a possibility that some of the number radio stations are operated by criminal syndicates who can broadcast from "state-within-a-state" areas, which are not controlled by the governments.

There are strong arguments in favor of the connection between the most well documented number radio stations and certain governments, especially during the Cold War and 90's. Most of the well known number radio stations from that times have been discovered due to their broadcasts efficiency and persistency (stable schedule, intensive and well directed signal that is easy to detect). Running efficient number station with great coverage, always requires very experienced staff with deep knowledge in shortwave signal propagation, and significant budget (to build effective transmission site at the right place). Such kind of assets are unlikely to become in possession of radio amateurs and enthusiasts, willing to create a prank and conspiracy. For instance, the members of the technical staff who work for the owner of the number station, must be capable of doing some unique research and tests in advance (before even start building the broadcast station). First, they have to search and locate short wave frequencies that are of interest (useful for the signal propagation, located outside the short wave broadcasting bands), and not being actively exploited by anyone else. Once those frequencies are finally located, they need to become "occupied" by running beacons or test broadcasts. Later, the staff might need to repeat that procedure again and again if the currently occupied frequencies become actively jammed or some long-lasting interference occurred. Note that the aim of the occupation is to "mark" the frequency already taken and prevent someone else from using it. Having the required frequencies located and occupied, the technical staff should concentrate its efforts in direction to solve the most difficult problem - the parameterization. The parameterization requires to have the output power, the antenna type, its direction, and the signal propagation properly estimated in advance, to provide reliable reception in the are where the listeners are located. If the numeric radio station goal is to cover with strong signal different areas, far from each other, some very expensive set of special antennas and transmitters should be used.

There are some specific details, related to the number broadcast stations, that somehow remain not discussed enough. Suppose the number radio stations are really devices for sending encrypted messages to the field agents. How do the field agents receive the broadcasts without rising suspicion? It is quite clear that possessing specially constructed highly sensitive shortwave radio receiver, equipped with directed outdoor antenna, is a compromise. Therefore, to remain anonymous, the agents should use very standard shortwave radio receiver, which anyone around can buy and possess. It is true that nowadays almost no one buys and keeps shortwave radio receiver, because the modern communications are digital and based on Internet, but during the Cold War the radio receivers were something that every family has in their possession. But the suggested use of standard shortwave radio receivers rises the question "how do the number radio station operators were able to create strong signal in the area where the agents were located". After all, if the signal strength is low, then the agents cannot hear well the messages and having number radio station broadcast is pointless. The answer is that the most number radio stations transmissions are well directed to cover with strong signal certain areas. In those areas the strong signal is easy to intercept with a standard radio receiver. One can see how different antennas for shortwave broadcast support (presumably installed at the transmission site), can create different type signal propagation and covering different areas, even if the position of the transmission site remains the same:

One can see in all those diagrams linked above, that all areas covered with high signal strength are really wide and it is not possible to use the area shape and position to suggest where exactly are the target listeners (the field agents). Depending on the conditions and the presence of jamming, more than one transmission site could be needed to sustain the schedule of the broadcasts and the coverage. To run number station regularly successfully, a feedback containing information regarding the quality of reception close to the target audience, is required. That kind of information allows to correct adequately and fast the output power, to change the type of antenna or its direction, and select the most proper frequency, in attempt to increase the reception quality in the area of interest. The feedback could also discover the presence of radio jamming. Since the target listeners by default have to remain undercover, they might not participate in the process of submitting the feedback. Therefore, the feedback should be provided by someone who is in the area of reception, but does not have an easy to trace connection to the target listeners and unlikely to be arrested and questioned. During the Cold War, the easiest and classical way to receive a feedback of that kind was through the staff at the embassies and consulates, located in the area of reception. The feedback should be combined with some data from chirp signal reception (see also "R-S-T-systems") to adjust the broadcasts. Such a feedback procedure also points to government backed organizations. It rarely could be handle by a private enterprise.

Some of the active number radio station transmitters have been successfully tracked by shortwave enthusiasts and their locations became revealed, by implementing well planned procedure of "T-hunting". Locating a shortwave transmitter which is far away from the receiver, generally requires three or more operational high-sensitive shortwave receivers, equipped with input signal level meter, all positioned as far as possible away from each other (but not too far from the transmitter), and connected to outdoor directional antennas, which direction can be (manually or automatically) changed. Ideally, the positions of the receiver should "surround" the hunted transmitter. Once the number radio station broadcast is detected, the operators of the receivers set them to listen to that broadcast and start changing the direction of the antennas, until the best reception (highest possible input signal level) is achieved. That might be a slow process. Finally, all the directions detected by the receivers, should to be projected together on a map in attempt to plot the area where the transmitter is located onto. The one creating the plot should be perfectly aware that the Earth has elliptic shape, and plot the direction vectors as curves using the Earth surface curvature profile, not as straight lines, unless the transmitter is located less than several hundred kilometers away from the receivers. Note that when performing a shortwave T-hunting, the more receivers are involved into the hunting, the more precise the location of the area where the transmitter is positioned is estimated. Someone might argue that two receivers are perfectly enough for tracing the transmitter location, since the Law of cosines requires only two angles (two directions) and the distance between the locations where the angles are measured. But even if such a statement is correct mathematically, it is totally idealistic and does not take into account the complexity of the signal propagation (find details here, and here). The lower is the frequency used by the hunted transmitter, the higher is the location estimation error, the more signal direction measures at different locations are required to minimize the transmitter position variance.

The "Lincolnshire poacher" number station transmitter became successfully T-hunted. Its transmission site was located inside the area where RAF air base Akrotiri is. Because the air base area is controlled by the British army, one might suggest that the "Lincolnshire poacher" was most probably an essential part of the infrastructure used by MI6 for sending messages to the British agents and representatives in Europe, Middle East, and Africa. "Lincolnshire poacher" is not operational anymore (seems like even the antennas were removed).

The exact locations of most number radio stations transmission sites, built and exploited during the Cold War years, are unknown. Those sites were closed in early 90's when the shortwave radio enthusiasts did not have the level of coordination for T-hunting. Apart from documenting the broadcasts (schedule, audio records, the observed signal level), the shortwave enthusiasts did not do locate precisely the transmission sites. They knew only which country a particular broadcast is coming from: Federal Republic of Germany, German Democratic Republic (the Eastern Germany before the unification of Germany), Italy, Poland, Czech Republic, Bulgaria, Hungary, USSR. Taiwan is found to be the Asian country with longest tradition in running voice broadcasting number radio stations, mainly during 80's and 90's, and occasionally even today, but their activity is not well documented (their area of operations was and remains Mainland China and their transmission direction and output power are dedicated to service only that specific area). It is unthinkable for an independent private enterprise to run and operate a number radio station in any of the countries listed above before or after the Cold War, which is in favor of the hypothesis that most of the number radio stations are run by state-owned agencies.

One of the previous location of the famous UVB-76 "buzzer" number radio station transmitters has been discovered by T-hunting. Some of the Russian shortwave enthusiasts, involved into the hunting, managed to visit the building, used before to host the transmitter and antennas. They found there abandoned soviet military radio equipment and logs. Some of the currently active UVB-76 transmission sites are also located and reported. It seems like UCB-76 uses more than two transmission sites.

After the unification of Germany, a large number of speech generators were found abandoned inside the communication centers used by Stasi. By comparing the speech they produce to the one found in the audio records documenting the activity of the East German number radio stations during the Cold War, it is easy to uncover Stasi as number radio stations operator.

4. Whom the number radio stations broadcast to

It is explained above why the intelligence services and military are the most suspected organizations responsible for running most of the known number radio stations. In case that explanation is true, the target listeners of the number radio station broadcasts are agents or military units. If the hypothesis that some of the number radio stations are run by drug cartels, is true, there audience might consist of drug dealers (suppliers) or drug producers.

Some of the target listeners must be very experienced. Some of the number radio stations use to broadcast by using Single-side band modulation (SSB), either LSB or USB, to effectively extend the coverage of the broadcasts. Starting from 80's, some of the advanced shortwave radio receivers has been shipper with modules for receiving SSB broadcast. But even having the module built-in, some experience in the tuning is required. Due to the lack of evidences, it not clear who is the target listeners of the SSB radio broadcast, transmitted by some of the number radio stations. For some of the field agents having receiver with SSB reception might be dangerous. One other side, if the target listener is an embassy, military unit, or some large enterprise, it is natural for them to be able to deal with receiving SSB broadcasts. Note that to receive SSB without obtaining special receiver, it is possible to use any regular shortwave receiver in combination with low-power and very simple shortwave transmitter, used to cause an interference. The transmitter does not need to be incorporated into the circuit of the receiver. It should be used as a stand-alone device. It might be used by field agents.

5. What kind of communication protocol do the number stations use

This is the most intriguing question when it comes to explain what do the voice messages really contain and what kind of protocol was used for hiding the real content. Unfortunately, no information regarding the content of the messages has ever been made public. Still, one can make a good guess how the messages are being encrypted and how could they be decrypted. Of course, without having the decryption key, even if the protocol for composing the messages is known in details, their content will remain unknown. The good news is that almost everyone good in finding, tracing, analyzing, and reproducing data modes, might be perfectly capable to guess the protocol implemented for creating the encrypted messages and mimic it.

To make their communication unbreakable, the number radio station operators have to use encryption key which is different for every message and cannot be related to the clear text. The most secure way to do that is to employ the one-time pad (OTP) based encryption. By using a pad the clear text is encrypted letter by letter (including spaces, commas, and other kind of "special" symbols). The result is an encrypted message block consists of a sequences (in groups) of numbers or/and letters. Without having the same pad at the receiver, those groups cannot be turned back into the original clear text of the message, even theoretically. If it is implemented properly, the OTP based method for message encryption does not create patterns in the encrypted text and it is resistant to any kind of known crypto analysis.

Different type of content requires different symbol sets, called "alphabets". For instance, a given alphabet for encrypting standard text messages can contain all English letters, the digits from 0 to 9, empty spaces, special symbols (like ".", ",", "/", "#", "$", and many more). If the content to encrypt contains only numbers, an alphabet containing mostly numbers shoyld be implemented. The symbols are randomly taken from the alphabet sets to compose an OTP. Any collection of one-time pads is called "code book". Every used symbol in the pad have to be marked properly, so it cannot be used twice. After the encrypted message is created, the pad used for encrypting the content needs to be destroyed immediately and effectively.

Sometimes it is not possible to use carefully generated OTP. For example, if the field agent cannot carry or receive regularly code books, the messages can be encrypted and decrypted by using as a pad some very usual book (like "The Hound of The Baskervilles" or some cook recipe book), that can be found in every book store of library. By using this method, every symbols of the message is represented (encrypted) as a sequence of three numbers: (i) the number of the page, (ii) the number of the row, (iii) the number of the position in the row of the symbol to encrypt. The number radio station operator and the field agent have to agree in advance on which book to use (title, edition, publisher).

To make the encrypted messages impossible to analyze by any third party organization, all transmitted messages have the same size. That means that inside the message there is a specific combination of symbols saying "here is the end of the message", so all digits of the message from that position to the end (to the fixes size), are just some random integer numbers or encrypted empty spaces, taken from the currently used pad. In addition to that and to prevent even guessing if the message content is related to some event, the broadcasts need to be made regular, based on specific schedule which never changes. In this case some or most of the messages should be sent empty, but only those having the decryption key can prove that and skip the decryption of the entire messages.

Given bellow is an example illustrating the use of code book and OTP.

Example: Creating an encrypted message of a fixed length

Let us encrypt the clear text message:

Hello World 1 2 3

by using the code book of 12 pads, message block length of 30 4-digit groups, provided in Appendix A, and adopting "/=" as a string terminating the actual message text flow. All groups describing a symbol positioned after the termination string, are not considered part of the message.

The clear text encryption is organized as follows:

  1. Choose a pad containing (at least once) the sought symbol.
  2. Randomly select one of the double-digit pad ID numbers printed bellow the pad.
  3. Select the cell containing the sought symbol inside the pad table, take the corresponding row and column number.
  4. Compose the group by concatenating the pad ID number selected in (2), and the numbers of row and column, taken in (3). Do not use the same group again while using the same code book.

One can perform step (4) the other way round - to put first the row and column numbers, and the pad ID number afterwards.

The result is a sequence of 17 4-digit groups:

HelloWorld123/=
8369273877412803249317510116862010661941661904801835086192670106

Another 13 groups of 4-digit random integer numbers need to be appended to this message to match the required message length of 30 groups:

836927387741280324931751
011686201066194166190480
183508619267010642340293
033292832203704372738001
243700429417520741004922
138940445672121109694742

Note that the groups 4234, 0293, 0332, 9283, 2203, 7043, 7273, 8001, 2437, 0042, 9417, 5207, 4100, 4922, 1389, 4044, 5672, 1211, 0969, 4742 are just randomly generated integer numbers (they are not based on the code book pad content and they are not part of the plain text message).

At this point the message groups composed above can be broadcasted by the operator of the numeric radio station.

If you are interested in constructing a small prototype of a number station to broadcast the encrypted message, this video tutorial explains how to do so by using Raspberry Pi device and Python application.

Appendix A. The synthetic alphabet and the code book

Table A1. The synthetic alphabet providing symbol space for generating the pads in the code book. All empty cells in the tables are also part of the alphabet. The goal of repeating given symbol in the alphabet is to distort the probability to repeatedly insert that particular symbol in the pads. Note that the symbols "█" are not part of the alphabet.

abcdefghijklm
nopqrstuvwxyz
ABCDEFGHIJKLM
NMPQRSTUVWXYZ
abcdefghijklm
nopqrstuvwxyz
ABCDEFGHIJKLM
NMPQRSTUVWXYZ
 
,,,,,,,,.
.....++++----
////****%%%%#
###@@@@====((
(())))0123456
789

 

Table A2. Code book of 12 pads. Each pad contains 100 symbols. The symbols are randomly selected from the synthetic alphabet symbol set given in Table A1.
Pad number: 01
0123456789
0q)FR#b)m.J
1r=)hp+ / d
2v i9vS)SH*
3(XgSke bvE
4a+J@fZ(@-r
5 %J5H@L#0
6 m( %(+(#,
7uuRs//tS)8
8WQ q9QO#k
92dH/K)q) /
ID: 74 50 30 21 66 37 23 00
Pad number: 02
0123456789
0(tODO .w#.
1Sxqr,OyM@p
2Z4Psj84rn-
3GIR.G,cuqO
4,I 6%@.dxT
5B*b*M,O /(
6P%) - rS#-
7PW/-x(hox%
81 H47,eBz9
9bGdp9z Z/a
ID: 04 39 55 51 38 03 72 32 10
Pad number: 03
0123456789
0@) (+Qwe N
1*5zu@ j#6%
2S%#Y)Tl7 H
3pv+/ amer
4.X- iHYy.+
5)=V.) f#N,
6ig#oe+U/-P
7w-v+s+Ib@E
8/89)l @YQ
9,**.)fL2+
ID: 11 84 36 92 89 52 27 97
Pad number: 04
0123456789
0T,.Ha, qi@
1TZiSzL@ QZ
2a==j,#6l,,
3qj)Vv --EF
4=l7E pfJA
5J )ZL3(=K)
6/. 2U.D.Z-
7L.fFP.-*nB
8r#,L +uwlA
9@vxG,,TmN)
ID: 95 77 19 42 88 16 46 96
Pad number: 05
0123456789
0-(Jle%/,Ki
1,yYz J qM
2JNpZ* FBh*
3 N%HpZ@c(z
40,Y=#(N)vB
5NQZ(+%EMoy
6 -#H+AA,)N
7+,O/vu f *
8) .FcdDvc,
9V Yj@JHanj
ID: 67 26 87 56 68 33 28 79
Pad number: 06
0123456789
0+W4T)Nq+f5
1)g( #w .qY
2 D jvJB *
3.-jDnYJb38
4s/)ISzBua)
51YTtY8 ny)
6p,VUI- mWf
7T gQqKRpwZ
8t+b(8f8htN
9S3)p )Pz()
ID: 69 48 82 41 91 02 20 99 75
Pad number: 07
0123456789
0Ya,dq).7%G
1 =A, 5=qNe
2oMcpi%JH--
3wbjFL2(WE0
4iJv4xl-n,H
5* ZG apVyA
6/y,* U)G
7j%qHN,+/-S
8qI/TS N +
9) s/F aOs+
ID: 86 85 47 35 90 78 73 18
Pad number: 08
0123456789
0@H5mq=h# J
1 /U*B p mJ
2brWQUpt8*k
3C-/yL%(i I
4yFU. tDQVy
5suYFV q@z
6s3 (KcdIHT
7,.Z@(sGuFy
8TWhS+VL#,
94qPyuH,A.V
ID: 34 98 15 08 53 13 58 09 81
Pad number: 09
0123456789
0@-C@c =p.e
1atq l Wu+A
2 @d YXtYOb
3bm5IJnzUg+
4YWRW9 (IJ,
5q d,qfI-,
6-(,B#.v/yH
7qE.Got e/j
8+.Wj,J d d
9aB)9LmN#vE
ID: 71 01 25 17 64 22 07 83 93
Pad number: 10
0123456789
0qLG =m.+B.
1ii2 7)tsog
2 4+dJa)-H.
3Qh@bI-, vp
4/(.,(qv @x
5g+=-%@b=S+
6waui.q((*+
74+zrAmiwsl
8#H -@DErG
9gp I,,nc-L
ID: 45 12 06 63 59 65 49 14
Pad number: 11
0123456789
0 vQ5RKoL8/
1%eb+ZTiIPd
2r,YizGC- *
3Unm=z,=e*l
4 +RSTFRhG/
5G.FB -CaNo
6v(W +OSdQ,
7VSGM.qA+q.
8V%W#nyH.-O
9*pc LP,em=
ID: 61 60 54 05 94 44 57 43
Pad number: 12
0123456789
05I. q+j#f,
1q,IM=Fpvkb
2z lpSwdiMF
3u=%.WfVR
4ArFYL#Y i+
5W.s*/rp+fE
6 hfvJFZqdx
7s)pb()JA@E
8XQ%E, Qzx=
9-aBoI#9/gv
ID: 24 62 31 76 80 40 29 70

Sunday, 6 May 2018

Installing Intel Python 3, tensorflow-gpu, and multiple versions of CUDA and cuDNN on CentOS 7

Content:

1. Introduction

2. Enabling the use of EPEL repository

3. Installing multiple CUDA versions on CentOS 7

4. Monitoring the NVidia GPU device by nvidia-smi

5. Making the software state in the NVIDIA driver persistent

6. Installing cuDNN for multiple versions of CUDA

7. Installing Intel Python 3 and tensorflow-gpu

8. Testing the CUDA and cuDNN installation

8.1. Testing if cuDNN library is loadable

8.2. Testing the CUDA Python 3 integration by using Numba

8.3. Testing the CUDA Python 3 integration by using tensorflow-gpu


This publication describes how to install multiple versions of CUDA and cuDNN on the same system running CentOS 7 to support various applications, and tensorflow in particular (via tensorflow-gpu). The recipes provided bellow can be follow when adding GPU computing support for compute nodes, which are part of HPC cluster.


This is an optional step, applicable if the configuration for using EPEL repository is not presented in /etc/yum.repos.d. EPEL is required here because the installation of the nvidia graphics driver, part of CUDA packages, requires the presence of DKMS in the system in advance. That package is included in EPEL. To use EPEL install first its repository package:

# yum install epel-release
# yum update

The dkms RPM package will be installed later, as a dependence required by the CUDA packages (see next section).


The most reasonable question here is why do we need multiple version of CUDA installed and supported locally on the system. Its answer is straightforward - it is all about the application software specific requirements. Some software products are very specific about the version of CUDA.

The most rational way to install the CUDA packages on CentOS 7 is through yum. NVidia provides the configuration files for using their yum repositories as a separate RPM package, which might be downloaded here:

https://developer.nvidia.com/cuda-downloads

To initiate the download consequently select Linux > x86_64 > CentOS > 7 > rpm (network) > Download as shown in the screen shots bellow:


and installed by following the instructions given bellow the "Download" button.

From time to time some inconsistencies appear in the CUDA yum repository. To prevent any problems they might cause edit the file /etc/yum.repos.d/cuda.repo by changing there the line:

enabled=1

into

enabled=0

From now on, every time an access to the CUDA repository RPM packages is required, do supply the command line option --enablerepo=cuda to yum.

After finishing with the yum configuration install the RPM packages containing the versions of CUDA currently supported by the vendor:

# yum --enablerepo=cuda install cuda-8-0 cuda-9-0 cuda-9-1

That will install plenty of packages. Take into account their installation size and prepare to meet that demand for disk space.

If, by any chance, the installer misses to install the packages nvidia-kmod, xorg-x11-drv-nvidia, xorg-x11-drv-nvidia-libs, and xorg-x11-drv-nvidia-gl, install them separately:

# yum --enablerepo=cuda install nvidia-kmod xorg-x11-drv-nvidia xorg-x11-drv-nvidia-libs xorg-x11-drv-nvidia-gl

The tool nvidia-smi is part of the package xorg-x11-drv-nvidia. It shows the current status of the NVidia GPU device:

$ nvidia-smi

Sun May  6 17:15:10 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.30                 Driver Version: 390.30                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Quadro K620         On   | 00000000:02:00.0 Off |                  N/A |
| 34%   36C    P8     1W /  30W |      1MiB /  2000MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

That tool is useful to check how many applications are currently running on the GPU device, what is the temperature there, the consumed power, the utilization rate, and what amount of memory is taken by the applications.


To prevent the driver from releasing the NVidia GPU device, when that device is not in use by any process, the daemon nvidia-persistenced (part of the package xorg-x11-drv-nvidia) needs to be enabled and started:

# systemctl enable nvidia-persistenced
# systemctl start nvidia-persistenced

The cuDNN library and header files can be downloaded from the web page of the vendor at:

https://developer.nvidia.com/cudnn

Note that a proper user registration is required to obtain the cuDNN files. Also, you need to download the archives with cuDNN library and header files for each and every CUDA version locally installed and supported. Which process, in turn, will end up bringing the following files into the download directory:

cudnn-8.0-linux-x64-v6.0.tgz
cudnn-9.0-linux-x64-v7.tgz
cudnn-9.1-linux-x64-v7.tgz

To proceed with the installation, unpack the content of the archives into the respective CUDA installation folders and recreate the database with the dynamic linker run time bindings, by executing (as root or super user) the command lines:

# tar --strip-components 1 -xf cudnn-8.0-linux-x64-v6.0.tgz -C /usr/local/cuda-8.0
# tar --strip-components 1 -xf cudnn-9.0-linux-x64-v7.tgz -C /usr/local/cuda-9.0
# tar --strip-components 1 -xf cudnn-9.1-linux-x64-v7.tgz -C /usr/local/cuda-9.1
# ldconfig /

It is recommended to check the successful archive unpacking and the proper recreation of the database with the dynamic linker run time bindings, by listing the database cache and grep the output for locating the string "cudnn" in it:

$ ldconfig -p | grep cudnn

The grep result indicating successful cuDNN installation, will look like:

libcudnn.so.7 (libc6,x86-64) => /usr/local/cuda-9.0/targets/x86_64-linux/lib/libcudnn.so.7
libcudnn.so.7 (libc6,x86-64) => /usr/local/cuda-9.1/targets/x86_64-linux/lib/libcudnn.so.7
libcudnn.so.6 (libc6,x86-64) => /usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudnn.so.6
libcudnn.so (libc6,x86-64) => /usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudnn.so
libcudnn.so (libc6,x86-64) => /usr/local/cuda-9.0/targets/x86_64-linux/lib/libcudnn.so
libcudnn.so (libc6,x86-64) => /usr/local/cuda-9.1/targets/x86_64-linux/lib/libcudnn.so

Do not become confused due to the multiple declarations made for libcudnn.so in the database (as seen in the output above). Seemingly, that indicates a collision, but note that each of libcudnn.so files is a symlink and it also provides an unique version number. That number is used by the tensorflow libraries to find which of the files matches best the version requirements.


If Intel Python 3 is not available in the system, follow the instructions given here:

https://software.intel.com/en-us/articles/installing-intel-free-libs-and-python-yum-repo

on how to install it. It is a single RPM package (mind its large installation size of several gigabytes) which contains tensorflow but (currently) does not include tensorflow-gpu module. Once Intel Python 3 is available the tensorflow-gpu module could be installed by invoking pip (the one provided by Intel Python 3).

Do not install tensorflow-gpu or any other module for Intel Python 3 as root or super user. Avoid any module installations inside the /opt/intel/intelpython3/ folder. Instead, perform the installation as unprivileged user and append the --user option to pip:

$ /opt/intel/intelpython3/bin/pip install --user tensorflow-gpu

The output information generated during the installation process should look like:

Collecting tensorflow-gpu
  Downloading https://files.pythonhosted.org/packages/59/41/ba6ac9b63c5bfb90377784e29c4f4c478c74f53e020fa56237c939674f2d/tensorflow_gpu-1.8.0-cp36-cp36m-manylinux1_x86_64.whl (216.2MB)
    100% |████████████████████████████████| 216.3MB 7.8kB/s 
Collecting protobuf>=3.4.0 (from tensorflow-gpu)
  Downloading https://files.pythonhosted.org/packages/74/ad/ecd865eb1ba1ff7f6bd6bcb731a89d55bc0450ced8d457ed2d167c7b8d5f/protobuf-3.5.2.post1-cp36-cp36m-manylinux1_x86_64.whl (6.4MB)
    100% |████████████████████████████████| 6.4MB 266kB/s 
Collecting gast>=0.2.0 (from tensorflow-gpu)
  Downloading https://files.pythonhosted.org/packages/5c/78/ff794fcae2ce8aa6323e789d1f8b3b7765f601e7702726f430e814822b96/gast-0.2.0.tar.gz
Collecting termcolor>=1.1.0 (from tensorflow-gpu)
  Downloading https://files.pythonhosted.org/packages/8a/48/a76be51647d0eb9f10e2a4511bf3ffb8cc1e6b14e9e4fab46173aa79f981/termcolor-1.1.0.tar.gz
Requirement already satisfied: wheel>=0.26 in /opt/intel/intelpython3/lib/python3.6/site-packages (from tensorflow-gpu)
Collecting tensorboard<1.9.0,>=1.8.0 (from tensorflow-gpu)
  Downloading https://files.pythonhosted.org/packages/59/a6/0ae6092b7542cfedba6b2a1c9b8dceaf278238c39484f3ba03b03f07803c/tensorboard-1.8.0-py3-none-any.whl (3.1MB)
    100% |████████████████████████████████| 3.1MB 545kB/s 
Collecting grpcio>=1.8.6 (from tensorflow-gpu)
  Downloading https://files.pythonhosted.org/packages/c8/b8/00e703183b7ae5e02f161dafacdfa8edbd7234cb7434aef00f126a3a511e/grpcio-1.11.0-cp36-cp36m-manylinux1_x86_64.whl (8.8MB)
    100% |████████████████████████████████| 8.8MB 195kB/s 
Collecting astor>=0.6.0 (from tensorflow-gpu)
  Downloading https://files.pythonhosted.org/packages/b2/91/cc9805f1ff7b49f620136b3a7ca26f6a1be2ed424606804b0fbcf499f712/astor-0.6.2-py2.py3-none-any.whl
Requirement already satisfied: numpy>=1.13.3 in /opt/intel/intelpython3/lib/python3.6/site-packages (from tensorflow-gpu)
Requirement already satisfied: six>=1.10.0 in /opt/intel/intelpython3/lib/python3.6/site-packages (from tensorflow-gpu)
Collecting absl-py>=0.1.6 (from tensorflow-gpu)
  Downloading https://files.pythonhosted.org/packages/90/6b/ba04a9fe6aefa56adafa6b9e0557b959e423c49950527139cb8651b0480b/absl-py-0.2.0.tar.gz (82kB)
    100% |████████████████████████████████| 92kB 8.8MB/s 
Requirement already satisfied: setuptools in /opt/intel/intelpython3/lib/python3.6/site-packages (from protobuf>=3.4.0->tensorflow-gpu)
Requirement already satisfied: werkzeug>=0.11.10 in /opt/intel/intelpython3/lib/python3.6/site-packages (from tensorboard<1.9.0,>=1.8.0->tensorflow-gpu)
Collecting bleach==1.5.0 (from tensorboard<1.9.0,>=1.8.0->tensorflow-gpu)
  Downloading https://files.pythonhosted.org/packages/33/70/86c5fec937ea4964184d4d6c4f0b9551564f821e1c3575907639036d9b90/bleach-1.5.0-py2.py3-none-any.whl
Collecting markdown>=2.6.8 (from tensorboard<1.9.0,>=1.8.0->tensorflow-gpu)
  Downloading https://files.pythonhosted.org/packages/6d/7d/488b90f470b96531a3f5788cf12a93332f543dbab13c423a5e7ce96a0493/Markdown-2.6.11-py2.py3-none-any.whl (78kB)
    100% |████████████████████████████████| 81kB 8.9MB/s 
Collecting html5lib==0.9999999 (from tensorboard<1.9.0,>=1.8.0->tensorflow-gpu)
  Downloading https://files.pythonhosted.org/packages/ae/ae/bcb60402c60932b32dfaf19bb53870b29eda2cd17551ba5639219fb5ebf9/html5lib-0.9999999.tar.gz (889kB)
    100% |████████████████████████████████| 890kB 1.7MB/s 
Building wheels for collected packages: gast, termcolor, absl-py, html5lib
  Running setup.py bdist_wheel for gast ... done
  Stored in directory: /home/vesso/.cache/pip/wheels/9a/1f/0e/3cde98113222b853e98fc0a8e9924480a3e25f1b4008cedb4f
  Running setup.py bdist_wheel for termcolor ... done
  Stored in directory: /home/vesso/.cache/pip/wheels/7c/06/54/bc84598ba1daf8f970247f550b175aaaee85f68b4b0c5ab2c6
  Running setup.py bdist_wheel for absl-py ... done
  Stored in directory: /home/vesso/.cache/pip/wheels/23/35/1d/48c0a173ca38690dd8dfccfa47ffc750db48f8989ed898455c
  Running setup.py bdist_wheel for html5lib ... done
  Stored in directory: /home/vesso/.cache/pip/wheels/50/ae/f9/d2b189788efcf61d1ee0e36045476735c838898eef1cad6e29
Successfully built gast termcolor absl-py html5lib
Installing collected packages: protobuf, gast, termcolor, html5lib, bleach, markdown, tensorboard, grpcio, astor, absl-py, tensorflow-gpu
Successfully installed absl-py-0.2.0 astor-0.6.2 bleach-1.5.0 gast-0.2.0 grpcio-1.11.0 html5lib-0.9999999 markdown-2.6.11 protobuf-3.5.2.post1 tensorboard-1.8.0 tensorflow-gpu-1.8.0 termcolor-1.1.0

NOTE: The files brought by the tensorflow-gpu installation to the local file system will be located under ${HOME}/.local/lib/python3.6/site-packages/ directory!

That kind of test is very easy to perform. If it returns no error that means all symbols brought by the library libcudnn.so are known to the Python 3 interpreter.

To perform the test create the Python 3 script:

import ctypes

t=ctypes.cdll.LoadLibrary("libcudnn.so")

print(t._name)

save it as a file under the name cudnn_loading_cheker.py and then execute the script:

$ /opt/intel/intelpython3/bin/python3 cudnn_loading_cheker.py

If the libcudnn.so is successfully loaded the script will return the name of the library file:

libcudnn.so

and rise an message otherwise.

Along with the other modules for scientific computing and data analysis, the Intel Python 3 package supplies Numba. To perform GPU computing based on CUDA, the Numba jit compiler requires the environmental variables NUMBAPRO_NVVM and NUMBAPRO_LIBDEVICE both properly declared before start compiling any Python code containing GPU instructions. Those variables should point to the installation tree of the latest version of CUDA:

$ export NUMBAPRO_NVVM=/usr/local/cuda-9.1/nvvm/lib64/libnvvm.so.3.2.0
$ export NUMBAPRO_LIBDEVICE=/usr/local/cuda-9.1/nvvm/libdevice

It is highly recommendable to declare these variables in ${HOME}/.bashrc file.

Once the variables are declared and loaded, execute the test script /opt/intel/intelpython3/lib/python3.6/site-packages/numba/cuda/tests/cudapy/test_matmul.py:

$ /opt/intel/intelpython3/bin/python3 /opt/intel/intelpython3/lib/python3.6/site-packages/numba/cuda/tests/cudapy/test_matmul.py

In case of successful execution the script will exit by displaying the message:

.
----------------------------------------------------------------------
Ran 1 test in 0.093s

OK

A simple script for testing tensorflow-gpu can be found here:

https://github.com/yaroslavvb/stuff/blob/master/matmul_benchmark.py

It should be downloaded and then executed by using Intel Python 3 interpreter:

$ /opt/intel/intelpython3/bin/python3 matmul_benchmark.py

and in case of successful execution the following result will appear on the screen:

/opt/intel/intelpython3/lib/python3.6/site-packages/h5py/__init__.py:34: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
2018-05-06 16:21:22.591713: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2018-05-06 16:21:22.684411: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:898] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-05-06 16:21:22.684824: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 0 with properties: 
name: Quadro K620 major: 5 minor: 0 memoryClockRate(GHz): 1.124
pciBusID: 0000:01:00.0
totalMemory: 1.95GiB freeMemory: 1.92GiB
2018-05-06 16:21:22.684855: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-05-06 16:21:23.151861: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-05-06 16:21:23.151903: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0 
2018-05-06 16:21:23.151916: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N 
2018-05-06 16:21:23.152061: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1692 MB memory) -> physical GPU (device: 0, name: Quadro K620, pci bus id: 0000:01:00.0, compute capability: 5.0)

 8192 x 8192 matmul took: 1.34 sec, 817.99 G ops/sec