FancyIndex - Better directory listing

Recently, I got my hands on a small server, with a big storage space.

I decided to use it as a lightweight archive file server, accessible over HTTPS.

So, let's try to do that today!

Searching available tools

I already know about a PHP/Js web-based file browser called h5ai. The design is quite nice but having a JS frontend is a huge constraint (remember, lightweight was a goal) and having to install and configure an entire PHP backend is bothersome.

If we try to check a bit the alternatives, we almost only get huge webapps, r/w file browsers etc., which is clearly too much for this need.

I chose NGINX as the web server, to easily handle HTTP/2, SSL etc.

So, let's check out the auto-indexing system of NGINX! It's generally the preferred way to do a web Read-only FS, and really fills its job, as it's the most minimalistic, simple and direct one available.

Still, the blank design and the contrast hurts the eyes, and the listing is a bit too small.

Let's try to see if NGINX have a way to customize that page!

The only thing I could find that would be "basic" was the Fancy index extension, but it doesn't look maintained, up to date and we don't even know if it's compatible, and if so, for how long.

Sooo, that's a no-no.

During this search, I thought about the compatibility issues: we don't want to tightly couple the web server and the directory listing printing, as, the day we'd move away from NGINX, we would have up to nothing new to handle, no matter the server.

For that reason, and following everything we found, let's roll our own!

The requirements

Let's make a small specification for what the project solution will do.

This tool is a small and lightweight server that prints a directory's content.

It uses the URL path and a configured base path to match directories.

If the directory cannot be found, it will return a custom 404 page.

The base path must exist and point to a valid folder. If not, at startup,
the server must crash with a clear error message showing that it's not valid.

Sounds good for a beginning.

Hhm... and what about the internal listing/file access logic?

Since this tool is made to run behind a reverse-proxy such as NGINX or Apache, as it only fills the role of directory-listing, the simplest configuration would follow the next logic:

Is the pointed path a valid file under the host's document root? yes -> serve the file, no -> continue
Proxy_pass to the server.

Looks perfect ! Let's add that to the spec.

Now that we have a clearer view on what to do and how it should act, let's think of what we actually want to use...

I eyed the Go language for some time, especially since:

it's a compiled language
the stdlib ships every tool to have a basic functional web server

Still, I never got the time to get a hang of it.

Well, a simple project is always the best way to start learning a language!

Let's go !

By a quick search, we find this tutorial, directly from the GoLang wiki.

If we look at what's proposed in this tutorial, we can see that, well... there's everything we would need to start learning!

Covered in this tutorial:

Creating a data structure with load and save methods

Using the net/http package to build web applications

Using the html/template package to process HTML templates

Using the regexp package to validate user input

Using closures

From the "Writing Web Applications" tutorial.

Now... For each request, the logic would be the following:

Grabbing the request URI
Constructing real system path by coupling a defined base path and the URI
If the resulting path doesn't point to a valid directory, return a 404
If the resulting path points to a valid directory, lists its content and returns a formatted list

The first thing we could do, to simplify our work but also to start learning about Go, is directory listing.

For now, let's try to write a function that lists the content of the working directory (pwd).

the ioutil standard lib have a ReadDir(string) ([]os.FileInfo, error) that is described as

ReadDir reads the directory named by dirname and returns a list of directory entries sorted by filename.

Looks perfect!

func handler() {
    dirs, err := ioutil.ReadDir(".")
    if err != nil {
        panic("Unable to read directory")
    }
}

Now that we have a way to cleanly list a directory content (and return a 404 if error == nil), we just need to iterate through the resulting array and just grab the bits of informations that we want.

Those informations are, separately, the folders and everything else: we want to be able to display all folders first.

Let's make a small struct to handle our result set!

Here, Files is used to describe every reachable entity that is not a folder, so symbolic links etc.

type Res struct {
    Path          string
    Folders       []string // For now, we'll want an array of folder names
    Files         []string // For now, we'll want an array of file names
}

func handler() {
    res := &Res{
        Path: ".",
        Folders:       []string{},
        Files:         []string{},
    }
    dirs, err := ioutil.ReadDir(res.Path)
    if err != nil {
        panic("Unable to read directory")
    }
}

That will hold everything together nicely.

We then can simply add a function that will iterate on each directory entry in dirs and, if it's a directory, add it to the Folders array, but else in the Files array.

type Res struct {
    Path    string
    Folders []string // For now, we'll want an array of folder names
    Files   []string // For now, we'll want an array of file names
}

func handler() {
    res := &Res{
        Path:    ".",
        Folders: []string{},
        Files:   []string{},
    }

    dirs, err := ioutil.ReadDir(res.Path)
    if err != nil {
        panic("Unable to read directory")
    }

    for _, f := range dirs {
        name := f.Name()

        if f.IsDir() {
            // If the directory is not named as "myDirectory/", add that final slash
            if !strings.HasSuffix(name, "/") {
                name += "/"
            }
            res.Folders = append(res.Folders, name)
        } else {
            res.Files = append(res.Files, name)
        }
    }
}

Simple, and that works !

Now, letting this rest a bit, let's try to play a bit with the HTTP server stuff and see what we can use.

If we look at the given example in the tutorial, we have this code to start with:

package main

import (
    "fmt"
    "net/http"
)

func handler(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "Hi there, I love %s!", r.URL.Path[1:])
}

func main() {
    http.HandleFunc("/", handler)
    http.ListenAndServe(":8080", nil)
}

Okay, so I guess handler is the actual request handler, as it have both a http.ResponseWriter struct and a http.Request struct (over a pointer).

By looking at the only instruction in the handler body, it looks like it's printing in a flux named w, which is our http.ResponseWriter variable, so I guess it's here I'd want to look into for organizing my directory listing.

But something catches the eye: the value passed in the print formatter: r.URL.Path[1:].

After investigation (Request, url), it seems like the entire URI is stored as an array of URL bits which, together, compose the URI.

That's neat, we'll keep that in memory, it will surely come in handy later!

Continuing our code reading, I see that this handler is effectively "mounted" to the route "/" (http.HandleFunc("/", handler)).

Nothing weird here.

If we try that example, something can immediately be noticed: whatever we type as a URI path ends up printed in the answer.

That means two things:

Every request that does not find a more precise or previously defined path is handled by the "/" handler.
We have, just here, 90% of our request handling! (Remember, the only thing we want is the path the user is trying to access).

I'll overlook the bits about the base path configuration and such.

Instead, we'll concentrate on the last thing we actually want: templates.

We want to be able to stylise as we want our directory listing, without a pain. Simple answer: HTML templates.

Does our magic tutorial have that? Yes!

The only thing we actually need to modify in our code is to add the template loading and change the Fprintf to the template engine's way of sending back content.

If we take a look at the given example,

<h1>Editing {{.Title}}</h1>

<form action="/save/{{.Title}}" method="POST">
<div><textarea name="body" rows="20" cols="80">{{printf "%s" .Body}}</textarea></div>
<div><input type="submit" value="Save"></div>
</form>

and the Go handler,

func editHandler(w http.ResponseWriter, r *http.Request) {
    title := r.URL.Path[len("/edit/"):]
    p, err := loadPage(title)
    if err != nil {
        p = &Page{Title: title}
    }
    t, _ := template.ParseFiles("edit.html")
    t.Execute(w, p)
}

Note that p is a struct defined a bit before this part, in the web tutorial.

type Page struct {
    Title string
Body  []byte
    }

We can easily adapt our code to use a HTML template !

func handler(w http.ResponseWriter, r *http.Request) {
    // res := &Res{}

    t, _ := template.ParseFiles("list.html")
    t.Execute(w, res)
}

I could've continued to read the guide further (and I'll actually do so for my next tweakings with Go), but as far as I'm concerned, we have everything we need right now !

So after a bit of prototyping, I can finally come to this source:

package main

import (
    "bytes"
    "fmt"
    "html/template"
    "io/ioutil"
    "net/http"
    "os"
    "strings"
    "time"

    config "./config"
)

// Res format of a folder's content
type Res struct {
    BasePath      string
    Path          string
    ParentPath    string
    IsNotRoot     bool
    Folders       []string // For now, we'll want an array of folder names
    Files         []string // For now, we'll want an array of file names
    Motd          string
    IsMotdEnabled bool
}

func parentPath(str string) string {
    if strings.Compare(str, "/") == 0 {
        return str
    }
    if strings.HasSuffix(str, "/") {
        str = str[:len(str)-1]
    }
    if strings.Count(str, "/") == 1 {
        return "/"
    }
    lio := strings.LastIndex(str, "/")
    return str[:lio]
}

func handler(w http.ResponseWriter, r *http.Request) {
    res := &Res{
        BasePath:      config.RootFSPath,
        Path:          r.URL.Path,
        Folders:       []string{},
        Files:         []string{},
        Motd:          config.Motd,
        IsMotdEnabled: config.MotdEnabled,
    }

    res.ParentPath = parentPath(res.Path)
    res.IsNotRoot = strings.Compare(res.Path, "/") != 0

    // Uniforming the end '/'
    if !strings.HasSuffix(res.Path, "/") {
        res.Path += "/"
    }

    var path bytes.Buffer

    path.WriteString(res.BasePath)
    path.WriteString(res.Path)

    dirs, err := ioutil.ReadDir(path.String())
    if err != nil {
        t, _ := template.ParseFiles("templates/404.html")
        t.Execute(w, res)
        return
    }
    for _, f := range dirs {
        name := f.Name()

        if f.IsDir() {
            if !strings.HasSuffix(name, "/") {
                name += "/"
            }
            res.Folders = append(res.Folders, name)
        } else {
            res.Files = append(res.Files, name)
        }
    }

    t, _ := template.ParseFiles("templates/list.html")

    t.Execute(w, res)
}

func main() {
    http.HandleFunc("/", handler)

    ListenURL := os.Getenv("HOST")
    if len(ListenURL) == 0 {
        ListenURL = "127.0.0.1:3001"
    }

    StartDate := time.Now().String()

    fmt.Printf(
        "Server starting at %s.\nListening to incoming requests on \"%s\"\n",
        StartDate, ListenURL)
    http.ListenAndServe(ListenURL, nil)
}

You'll also notice two additional things:

MOTD (which is basically a static text display that can be enabled and customized outside of the template itself).
A custom 404 page, too.

This server works great, handles requests blazingly fast and is dead simple to hack with!

I guess, as it's still beginner code, that a lot of things could be improved, but as far as I'm concerned, the tool is efficient as-is!