Transferring Data

Table of Contents

    IDUN has 2 dedicated servers for large data transfer:

    • idun-samba1.hpc.ntnu.no
    • idun-samba2.hpc.ntnu.no

    For small data transfer you can use IDUN login nodes or Open OnDemand web interface: https://apps.hpc.ntnu.no/

    The transfer of files to and from IDUN can be accomplished using network drives, scp, or sftp. The following sections exemplify the usage of these methods.

     

    Method 1 - WinSCP, FileZilla

    • WinSCP for Windows.
    • FileZilla for Windows, Mac and Linux

     

    WinSCP

    Download and install WinSCP application. Link:  https://winscp.net/eng/download.php

    Configuration:

    First time click YES:

    To copy file you can drag and drop files from left side to the right side:

     

    FileZilla

    Download link: https://filezilla-project.org/download.php?show_all=1

    Configuration:

    First time click OK:

    To copy file you can drag and drop files from left side to the right side:

     

    Method 2 - Network Drives - Mounting Idun's home and work directory to your local machine

    Idun has two samba servers ({idun-samba1,idun-samba2}.hpc.ntnu.no) that enable the mounting of a user's home and work directory directly to its local machine. The respective paths to the shares are:

    • Home directory: //<samba-server>/<username>
    • Work directory: //<samba-server>/work

    The following sections illustrate the mounting of a user's home directory for Windows, Linux, and macOS with idun-samba1.hpc.ntnu.no. For mounting the work directory, simply replace the home directory path with the work directory path in the examples.

     

    Windows

    In Windows, the File Explorer can be used to attach the home directory as a network drive. Navigate to File Explorer -> Map network drive and enter the details as shown below:

    Replace <username> with your Idun user. This mounts the user's home directory as drive Z into the File Explorer.

    usernames need to be entered in this form: WIN-NTNU-NO\<username>

     

    Linux

    In Linux, the mount.cifs command can be used to attach a user's home directory to a local directory. Write this on your Linux computer:

    $ sudo mount.cifs //idun-samba1.hpc.ntnu.no/<username> /tmp/mntpt -o username=<username>,domain=WIN-NTNU-NO

    In the above example, the home directory of <username> is attached to the local directory /tmp/mntpt. The original content of /tmp/mntpt is overshadowed with the files of the user's home directory from Idun. Files can now directly transferred to Idun by simply copying them to /tmp/mntpt.

    The umount command can be used to detach the home directory:

    $ sudo umount /tmp/mntpt

    This reveals the original content of /tmp/mntpt again. A script to simplify this process can be found here.

     

    macOS

    In macOS, Finder can be used to attach the home directory as a network drive. Navigate to Finder -> Go -> Connect to Server and enter the details as shown below:

    Replace <username> with your Idun user.

     

    Smbclient - Accessing NTNU's network drives directly from Idun

    Copying data to Idun by mounting the home and work directory from Idun directly to a local machine is acceptable if the data is located on the local machine, but it creates a lot of overhead if the data resides on another machine. For example, a user would need to perform the following steps to transfer data from NTNU's home directory to Idun's home directory:

    1. Mount NTNU's home directory to the local machine.
    2. Transfer the data from NTNU's home directory to the local machine.
    3. Mount Idun's home directory to the local machine.
    4. Transfer the data from the local machine to Idun's home directory.

    This transfers the data twice, once from NTNU's home directory to the local machine and once from the local machine to Idun. A better approach would be to directly transfer the data from NTNU's home directory to Idun's home directory, cutting the network traffic and data transfer time in half. The smbclient tool is an interactive program that enables the transferal of data directly between Idun and any of NTNU's samba shares, such as //home.ansatt.ntnu.no/<username>, //home.stud.ntnu.no/<username>, or //forskning.it.ntnu.no/ntnu/<share>.

    It can be invoked as follows:

    smbclient
    [<username>@idun-login1 ~]$ smbclient //home.ansatt.ntnu.no/<username> -U <username> -W WIN.NTNU.NO
Enter WIN.NTNU.NO\<username>'s password:
Domain=[WIN-NTNU-NO] OS=[SpinStream2] Server=[Windows 2000 Lan Manager]
smb: \> ls
  .                                   D        0  Mon Jan 14 10:26:49 2019
  ..                                  D        0  Wed Oct  9 17:52:59 2019
  public_html                         N       26  Fri Mar 22 17:35:01 2013
  .bash_profile                       A       31  Tue Jan  8 11:37:52 2013
  .bashrc                             A       25  Tue Jan  8 11:37:53 2013
  .emacs                              A       37  Tue Jan  8 11:37:53 2013
  .cache                              D        0  Tue Aug 27 12:54:05 2013
  .bash_history                       A     5265  Mon Jan 14 10:26:49 2019
  .ssh                                D        0  Wed Nov  8 10:29:47 2017
  .viminfo                            A     6639  Wed Nov  8 10:21:51 2017

smb: \> quit
[<username>@idun-login1 ~]$

    The above example connects to an employee's NTNU home directory, lists its content, and exits again.

    The help command lists all supported commands and can provide information about the usage of each command:

    smbclient-help
    [<username>@idun-login1 ~]$ smbclient //home.ansatt.ntnu.no/<username> -U <username> -W WIN.NTNU.NO
Enter WIN.NTNU.NO\<username>'s password:
Domain=[WIN-NTNU-NO] OS=[SpinStream2] Server=[Windows 2000 Lan Manager]
smb: \> help
?              allinfo        altname        archive        backup
blocksize      cancel         case_sensitive cd             chmod
chown          close          del            dir            du
echo           exit           get            getfacl        geteas
hardlink       help           history        iosize         lcd
link           lock           lowercase      ls             l
mask           md             mget           mkdir          more
mput           newer          notify         open           posix
posix_encrypt  posix_open     posix_mkdir    posix_rmdir    posix_unlink
posix_whoami   print          prompt         put            pwd
q              queue          quit           readlink       rd
recurse        reget          rename         reput          rm
rmdir          showacls       setea          setmode        scopy
stat           symlink        tar            tarmode        timeout
translate      unlock         volume         vuid           wdel
logon          listconnect    showconnect    tcon           tdis
tid            logoff         ..             !

smb: \> help cd
HELP cd:
        [directory] change/report the remote directory

smb: \>

    Useful commands for transferring data are:

    • ls: list the contents of the current remote directory
    • pwd: show current remote directory
    • cd <directory-name>: change the current remote directory to <directory-name>
    • lcd <directory-name>: change the current local directory to <directory-name>
    • get <remote-file> <local-file>: Copy remote file <remote-file> to <local-file>
    • mget <mask>: Copy all files that match <mask> from the share
    • put <local-file> <remote-file>: Copy local file <local-file> to <remote-file>
    • mput <mask>: Copy all files that match <mask> to the share
    • prompt [on|off]: Turn prompting for filenames for mget and mput on or off
    • recurse [on|off]: Turn directory recursion for mget and mput on or off

    The following example shows how to copy a remote folder to Idun:

    smbclient-example
    [<username>@idun-login1 ~]$ smbclient //home.ansatt.ntnu.no/<username> -U <username> -W WIN.NTNU.NO
Enter WIN.NTNU.NO\<username>'s password:
Domain=[WIN-NTNU-NO] OS=[SpinStream2] Server=[Windows 2000 Lan Manager]
smb: \> ls
  .                                   D        0  Fri Oct 11 12:04:05 2019
  ..                                  D        0  Wed Oct  9 17:52:59 2019
  copy-test                           D        0  Fri Oct 11 12:04:18 2019

smb: \> prompt off
smb: \> recurse on
smb: \> mget copy-test

    The above example shows the copying of the copy-test folder to the user's home folder on Idun. The user needs to turn off prompting in order to avoid being questioned for each file and turn on recurse in order to recursively copy the entire folder.

     

    SCP

    Secure Copy (scp) is a program to securely transfer files between computers. It uses the Secure Shell (ssh) for data transfer, providing the same authentication and security. For example, copying a file named myfile from a local system to Idun:

    $ scp myfile <username>@idun.hpc.ntnu.no:.
Password:

     

    Secure File Transfer Protocol

    Secure File Transfer Protocol (sftp) is a file transfer program, similar to ftp, which performs all operations over an encrypted ssh transport. Example, put file from local system to Idun:

    $ sftp idun.hpc.ntnu.no
Password:
Connected to idun.hpc.ntnu.no
sftp> put myfile

    For Windows users WinSCP is a free graphical SCP and SFTP client.

    Scroll to Top