SFTP directory listing

Added by Hajnal Akos about 6 years ago

Dear Developers,

I would like to list files (and get some details) in a directory accessible via sftp, and using jSAGA.
Probably, I am doing something wrong, but when I switch from http to sftp, the behaviour of
jSAGA becomes non-deterministic: sometimes I get correct results, in many cases, I get NoSuccessException,
and sometimes the code stucks at invoking blocking calls (such as close(), getSize(), getMTime()) without
terminating ever.

Can you please tell me what I am doing wrong? I used the code below that works for http but not for sftp.

Akos Hajnal

import java.text.SimpleDateFormat;
import java.util.Date;
import org.ogf.saga.context.Context;
import org.ogf.saga.context.ContextFactory;
import org.ogf.saga.file.File;
import org.ogf.saga.file.FileFactory;
import org.ogf.saga.namespace.NSDirectory;
import org.ogf.saga.namespace.NSFactory;
import org.ogf.saga.permissions.Permission;
import org.ogf.saga.session.Session;
import org.ogf.saga.session.SessionFactory;
import org.ogf.saga.url.URL;
import org.ogf.saga.url.URLFactory;

public class Ssh {

    public static void main(String[] args) throws Exception {

        // http works fine
        //URL url = URLFactory.createURL("http://grid.in2p3.fr/maven2/fr/in2p3/jsaga/jsaga-engine"); 
        //Session session = null; 

        URL url = URLFactory.createURL("sftp://");
        Session session = SessionFactory.createSession(false);
        Context ctx = ContextFactory.createContext("UserPass");
        ctx.setAttribute(Context.USERID, "root");
        ctx.setAttribute(Context.USERPASS, "****");

        NSDirectory dir = NSFactory.createNSDirectory(session, url);

        for (URL dirEntry: dir.list()) {
             if (dir.isEntry(dirEntry)) {
                 String fileName = dirEntry.getPath();

                 File f = FileFactory.createFile(session, URLFactory.createURL(dir.getURL().toString() + fileName));
                 long size = f.getSize();
                 long date = f.getMTime();
                 boolean readPermisssion = f.permissionsCheck("*", Permission.READ.getValue());
                 boolean writePermission = f.permissionsCheck("*", Permission.WRITE.getValue());
                 boolean executePermission = f.permissionsCheck("*", Permission.EXEC.getValue());

                         fileName + " " +
                         (readPermisssion ? "r" : "-") + (writePermission ? "w" : "-") + (executePermission ? "x" : "-") + " " + 
                         (date != 0l ? new SimpleDateFormat("dd-MMM-yyyy HH:mm").format(new Date(date)) : "?") + " " + 
                         (size != 0l ? "" + size + "B": "?"));

Replies (4)

RE: SFTP directory listing - Added by Hajnal Akos about 6 years ago

It seems that adding an extra parameter Flags.NONE.getValue() to call createFile solves the problem of NoSuccess exception:

File f = FileFactory.createFile(session, URLFactory.createURL(dir.getURL().toString() + fileName), Flags.NONE.getValue());

RE: SFTP directory listing - Added by Schwarz Lionel about 6 years ago

Hi Akos, it's a bug. Could you please open a new bug for this:
with the full stacktrace

RE: SFTP directory listing - Added by Schwarz Lionel about 6 years ago

It seems the problem comes from the underlying SSH library (com.jcraft.jsch) which is not thread-safe. I have tried with the latest JSCH version but got the same problem. There is no way for me to fix this in the JSAGA SSH adaptor, so I advise you to not use the READ flag if you just need the metadata. Your workaround is then the best thing to do now.

Using a non-thread-safe library is JSAGA is of course a concern for us, we think about moving to another library one day, which will hopefully avoid all these problems.


RE: SFTP directory listing - Added by Hajnal Akos about 6 years ago


I have created a bug report (#4474) with stack traces as requested. The thread-safety issue is a bad news for us: we are planning to serve many parallel users at the same time based on the jSAGA library. Simply making methods synchorized is not feasible, as we use many blocking calls inside high level methods, and servers may have very diverse response times.

We will see how it works in multi-threaded usage; maybe, we have to disable sftp.

Best regards,
Akos Hajnal