Docker源码学习 三

这篇文章继续来看docker的代码,这里我们重点看下daemon.NewDaemon的实现。

    registryService := registry.NewService(registryCfg)
    d, err := daemon.NewDaemon(daemonCfg, registryService)

NewDaemon的代码很长,我们一步一步来看下,首先:

    // Ensure we have compatible configuration options
    if err := checkConfigOptions(config); err != nil {
        return nil, err
    }
// checkConfigOptions checks for mutually incompatible config options
func checkConfigOptions(config *Config) error {
    // Check for mutually incompatible config options
    if config.Bridge.Iface != "" && config.Bridge.IP != "" {
        return fmt.Errorf("You specified -b & --bip, mutually exclusive options. Please specify only one.")
    }
    if !config.Bridge.EnableIPTables && !config.Bridge.InterContainerCommunication {
        return fmt.Errorf("You specified --iptables=false with --icc=false. ICC uses iptables to function. Please set --icc or --iptables to true.")
    }
    if !config.Bridge.EnableIPTables && config.Bridge.EnableIPMasq {
        config.Bridge.EnableIPMasq = false
    }
    return nil
}

这里检查了下输入的参数是否有矛盾的地方。接着:

    // Do we have a disabled network?
    config.DisableBridge = isBridgeNetworkDisabled(config) 
func isBridgeNetworkDisabled(config *Config) bool {
    return config.Bridge.Iface == disableNetworkBridge
}
......
const (
    defaultNetworkMtu    = 1500
    disableNetworkBridge = "none"
)

这里会设置config.DisableBridge这个参数,如果我们的配置文件中给出的Iface为none则表明我们disable了bridge network。继续看代码:

    // Verify the platform is supported as a daemon
    if runtime.GOOS != "linux" && runtime.GOOS != "windows" {
        return nil, ErrSystemNotSupported
    }

    // Validate platform-specific requirements
    if err := checkSystem(); err != nil {
        return nil, err
    }

这里会做一些平台相关的检查。继续看代码:

    // set up SIGUSR1 handler on Unix-like systems, or a Win32 global event
    // on Windows to dump Go routine stacks
    setupDumpStackTrap()
.......
func DumpStacks() {
    buf := make([]byte, 16384)
    buf = buf[:runtime.Stack(buf, true)]
    // Note that if the daemon is started with a less-verbose log-level than "info" (the default), the goroutine
    // traces won't show up in the log.
    logrus.Infof("=== BEGIN goroutine stack dump ===\n%s\n=== END goroutine stack dump ===", buf)
}

可以看到我们可以通过SIGUSR1这个信号量获取go routine的stack信息。后者是通过go提供的runtime模块实现的。继续看代码:

    // get the canonical path to the Docker root directory
    var realRoot string
    if _, err := os.Stat(config.Root); err != nil && os.IsNotExist(err) {
        realRoot = config.Root
    } else {
        realRoot, err = fileutils.ReadSymlinkedDirectory(config.Root)
        if err != nil {
            return nil, fmt.Errorf("Unable to get the full path to root (%s): %s", config.Root, err)
        }
    }
    config.Root = realRoot
    // Create the root directory if it doesn't exists
    if err := system.MkdirAll(config.Root, 0700); err != nil && !os.IsExist(err) {
        return nil, err
    }

这里会建立root目录。继续看代码:

    // set up the tmpDir to use a canonical path
    tmp, err := tempDir(config.Root)
    if err != nil {
        return nil, fmt.Errorf("Unable to get the TempDir under %s: %s", config.Root, err)
    }
    realTmp, err := fileutils.ReadSymlinkedDirectory(tmp)
    if err != nil {
        return nil, fmt.Errorf("Unable to get the full path to the TempDir (%s): %s", tmp, err)
    }
    os.Setenv("TMPDIR", realTmp)

这里建立了一个tmp目录。tempDir做的事情就是拼接出一个/root/tmp的目录,其中root路径由config.Root给出。继续看代码:

    // Set the default driver
    graphdriver.DefaultDriver = config.GraphDriver

这里设置了默认的graphdriver。根据http://www.infoq.com/cn/articles/docker-source-code-analysis-part1这篇文章我们可以知道graphdriver主要用于完成容器镜像的管理。driver的概念就是分了一层,熟悉面向对象的同学把其认为是实现某个接口的类就行了,不过docker这里会复杂一些,比如我的driver是基于XXX实现的,那么除了实现与XXX交互的各个方法外,docker在启动的时候还需要保证XXX这个服务正常运行。继续看代码:

    // Load storage driver
    driver, err := graphdriver.New(config.Root, config.GraphOptions)
    if err != nil {
        return nil, fmt.Errorf("error initializing graphdriver: %v", err)
    }
    logrus.Debugf("Using graph driver %s", driver)

    d := &Daemon{}
    d.driver = driver

这里可以看到docker加载了graphdriver的driver。我满来看下这里的New:

func New(root string, options []string) (driver Driver, err error) {
    for _, name := range []string{os.Getenv("DOCKER_DRIVER"), DefaultDriver} {
        if name != "" {
            logrus.Debugf("[graphdriver] trying provided driver %q", name) // so the logs show specified driver
            return GetDriver(name, root, options)
        }
    }

这里显示获取driver的列表,DefaultDriver现在使用的是配置中的config.GraphDriver。config.GraphDriver默认是””,所以我们不会执行if中的代码而是由docker猜测一个可能的driver:

    // Guess for prior driver
    priorDrivers := scanPriorDrivers(root)
    for _, name := range priority {
        if name == "vfs" {
            // don't use vfs even if there is state present.
            continue
        }
        for _, prior := range priorDrivers {
            // of the state found from prior drivers, check in order of our priority
            // which we would prefer
    ......
        // Check for priority drivers first
    for _, name := range priority {
        driver, err = GetDriver(name, root, options)
        if err != nil {
            if err == ErrNotSupported || err == ErrPrerequisites || err == ErrIncompatibleFS {
                continue
            }
            return nil, err
        }
        return driver, nil
    }

    // Check all registered drivers if no priority driver is found
    for _, initFunc := range drivers {
        if driver, err = initFunc(root, options); err != nil {
            if err == ErrNotSupported || err == ErrPrerequisites || err == ErrIncompatibleFS {
                continue
            }
            return nil, err
        }
        return driver, nil
    }
    return nil, fmt.Errorf("No supported storage backend found")

假设我们的driver选取了aufs,此时会执行GetDriver,看下其实现:

func GetDriver(name, home string, options []string) (Driver, error) {
    if initFunc, exists := drivers[name]; exists {
        return initFunc(filepath.Join(home, name), options)
    }
    logrus.Errorf("Failed to GetDriver graph %s %s", name, home)
    return nil, ErrNotSupported
}

可以看到这里调用了driver注册的initFunc函数来执行初始化。那么drivers中的内容是何时注册的呢?通过Register函数注册的:

func Register(name string, initFunc InitFunc) error {
    if _, exists := drivers[name]; exists {
        return fmt.Errorf("Name already registered %s", name)
    }   
    drivers[name] = initFunc

    return nil 
}

比如在aufs的代码中可以看到:

func init() {
    graphdriver.Register("aufs", Init)
}

现在问题来了,这里的init是什么时候被调用的呢?其实init方法是go的一个特性,其会的在main函数调用前调用。

继续看代码,这里来看下aufs的Init做了什么事情:

// New returns a new AUFS driver.
// An error is returned if AUFS is not supported.
func Init(root string, options []string) (graphdriver.Driver, error) {

    // Try to load the aufs kernel module
    if err := supportsAufs(); err != nil {
        return nil, graphdriver.ErrNotSupported
    }

    fsMagic, err := graphdriver.GetFSMagic(root)
    if err != nil {
        return nil, err
    }
    if fsName, ok := graphdriver.FsNames[fsMagic]; ok {
        backingFs = fsName
    }

    for _, magic := range incompatibleFsMagic {
        if fsMagic == magic {
            return nil, graphdriver.ErrIncompatibleFS
        }
    }

首先是检查系统是否支持aufs,如果支持则获取fs的magic number然后获取对应的fsName,这里就是aufs。然后检查下兼容性。接着:

    paths := []string{
        "mnt", 
        "diff",
        "layers",
    }

    a := &Driver{
        root:   root,
        active: make(map[string]int),
    }

    // Create the root aufs driver dir and return
    // if it already exists
    // If not populate the dir structure
    if err := os.MkdirAll(root, 0755); err != nil {
        if os.IsExist(err) {
            return a, nil
        }
        return nil, err
    }

    if err := mountpk.MakePrivate(root); err != nil {
        return nil, err
    }

    for _, p := range paths {
        if err := os.MkdirAll(path.Join(root, p), 0755); err != nil {
            return nil, err
        }
    }
    return a, nil
}

可以看到这里建立了aufs的root目录。我们之前将driver比作接口和实现类的关系,这里的&Driver中的Driver就是我们的接口,其需要实现下面几个方法:

// Driver is the interface for layered/snapshot file system drivers.
type Driver interface {
    ProtoDriver
    // Diff produces an archive of the changes between the specified
    // layer and its parent layer which may be "".
    Diff(id, parent string) (archive.Archive, error)
    // Changes produces a list of changes between the specified layer
    // and its parent layer. If parent is "", then all changes will be ADD changes.
    Changes(id, parent string) ([]archive.Change, error)
    // ApplyDiff extracts the changeset from the given diff into the
    // layer with the specified id and parent, returning the size of the
    // new layer in bytes.
    ApplyDiff(id, parent string, diff archive.ArchiveReader) (size int64, err error)
    // DiffSize calculates the changes between the specified id
    // and its parent and returns the size in bytes of the changes
    // relative to its base filesystem directory.
    DiffSize(id, parent string) (size int64, err error)
}

这几个方法的实现在具体的实现文件中可以找到,比如我们可以在aufs的实现中找到这些方法的实现。换句话说就是aufs实现了这个接口。

我们回到daemon.NewDaemon中来。现在我们 已经看到了graphdriver为aufs时候的初始化实现,主要就是设置了aufs需要的目录。然后我们继续看代码:

    d := &Daemon{}
    d.driver = driver
    // Ensure the graph driver is shutdown at a later point
    defer func() {
        if err != nil {
            if err := d.Shutdown(); err != nil {
                logrus.Error(err)
            }
        }
    }()

这里我们建立了一个Daemon的结构体,并将我们的graphdriver做为其driver属性,同时保证代码结束的时候会调用Shutdown。继续看代码:

    // Verify logging driver type
    if config.LogConfig.Type != "none" {
        if _, err := logger.GetLogDriver(config.LogConfig.Type); err != nil {
            return nil, fmt.Errorf("error finding the logging driver: %v", err)
        }
    }
    logrus.Debugf("Using default logging driver %s", config.LogConfig.Type)

这里检查了下logging driver。继续看代码:

    // Configure and validate the kernels security support
    if err := configureKernelSecuritySupport(config, d.driver.String()); err != nil {
        return nil, err
    }

这里检查了下内核的security方面的支持。继续看代码:

    daemonRepo := filepath.Join(config.Root, "containers")

    if err := system.MkdirAll(daemonRepo, 0700); err != nil && !os.IsExist(err) {
        return nil, err
    }

这里建立了daemon的repo的目录。继续看代码:

    // Migrate the container if it is aufs and aufs is enabled
    if err := migrateIfDownlevel(d.driver, config.Root); err != nil {
        return nil, err
    }

    logrus.Debug("Creating images graph")
    g, err := graph.NewGraph(filepath.Join(config.Root, "graph"), d.driver)
    if err != nil {
        return nil, err
    }

    // Configure the volumes driver
    if err := configureVolumes(config); err != nil {
        return nil, err
    }

我们看下configureVolumes,根据注释这里应该是建立了一个volume的driver:

func configureVolumes(config *Config) error {
    volumesDriver, err := local.New(config.Root)
    if err != nil {
        return err
    }
    volumedrivers.Register(volumesDriver, volumesDriver.Name())
    return nil
}

这里的实现其实和上面的graphdriver类似。我们继续看代码:

    trustKey, err := api.LoadOrCreateTrustKey(config.TrustKeyPath)
    if err != nil {
        return nil, err
    }

    trustDir := filepath.Join(config.Root, "trust")

    if err := system.MkdirAll(trustDir, 0700); err != nil && !os.IsExist(err) {
        return nil, err
    }
    trustService, err := trust.NewTrustStore(trustDir)
    if err != nil {
        return nil, fmt.Errorf("could not create trust store: %s", err)
    }

这里初始化了和秘钥相关的目录。继续看代码:

    eventsService := events.New()
    logrus.Debug("Creating repository list")
    tagCfg := &graph.TagStoreConfig{
        Graph:    g,
        Key:      trustKey,
        Registry: registryService,
        Events:   eventsService,
        Trust:    trustService,
    }
    repositories, err := graph.NewTagStore(filepath.Join(config.Root, "repositories-"+d.driver.String()), tagCfg)
    if err != nil {
        return nil, fmt.Errorf("Couldn't create Tag store: %s", err)
    }

这里简历了一个events,接着是建立了一个TagStore。目前我们都把他们看成是结构体就行了。继续看代码:

    d.netController, err = initNetworkController(config)
    if err != nil {
        return nil, fmt.Errorf("Error initializing network controller: %v", err)
    }

可以看到这里涉及到了d.netController,我们知道d就是我们的daemon,所以d的属性肯定是以后会被用到的东西。这里初始化了网络controller,来看下其实现:

func initNetworkController(config *Config) (libnetwork.NetworkController, error) {
    netOptions, err := networkOptions(config)
    if err != nil {
        return nil, err
    }

    controller, err := libnetwork.New(netOptions...)
    if err != nil {
        return nil, fmt.Errorf("error obtaining controller instance: %v", err)
    }

这里调用了libnetwork,libnetwork我们以后会重点研究。总之这里的代码负责初始化我们的网络环境。继续看代码:

    graphdbPath := filepath.Join(config.Root, "linkgraph.db")
    graph, err := graphdb.NewSqliteConn(graphdbPath)
    if err != nil {
        return nil, err
    }
    
    d.containerGraph = graph

这里简历了一个sqlite的db,并且也赋值给了daemon。继续看代码:

    var sysInitPath string
    if config.ExecDriver == "lxc" {
        initPath, err := configureSysInit(config)
        if err != nil {
            return nil, err
        }
        sysInitPath = initPath
    }

    sysInfo := sysinfo.New(false)
    // Check if Devices cgroup is mounted, it is hard requirement for container security,
    // on Linux/FreeBSD.
    if runtime.GOOS != "windows" && !sysInfo.CgroupDevicesEnabled {
        return nil, fmt.Errorf("Devices cgroup isn't mounted")
    }

这里会判断我们的系统是否支持cgroup。继续看代码:

    ed, err := execdrivers.NewDriver(config.ExecDriver, config.ExecOptions, config.ExecRoot, config.Root, sysInitPath, sysInfo)
    if err != nil {
        return nil, err
    }

这里其实和我们上面说的graph的driver初始化类似,也是根据参数调用对应的NewDriver函数,后者会在系统上初始化一些东西后返回一个driver,并且这个driver也实现了接口要求的方法。对于native来说这里使用的是libcontainer来获取driver的factory熟悉。相比以后的事情都是交给libcontainer来做吧。libcontainer和libnetwork我们以后会单独分析。继续看代码:

    d.ID = trustKey.PublicKey().KeyID()
    d.repository = daemonRepo
    d.containers = &contStore{s: make(map[string]*Container)}
    d.execCommands = newExecStore()
    d.graph = g
    d.repositories = repositories
    d.idIndex = truncindex.NewTruncIndex([]string{})
    d.sysInfo = sysInfo
    d.config = config 
    d.sysInitPath = sysInitPath
    d.execDriver = ed
    d.statsCollector = newStatsCollector(1 * time.Second)
    d.defaultLogConfig = config.LogConfig
    d.RegistryService = registryService
    d.EventsService = eventsService
    d.root = config.Root
    go d.execCommandGC()

    if err := d.restore(); err != nil {
        return nil, err
    }

    return d, nil
}

这里首先是将一大堆我们上面建立的结构体都扔给我们的daemon,然后调用d.execCommandGC()后台执行一些清理工作:

// execCommandGC runs a ticker to clean up the daemon references
// of exec configs that are no longer part of the container.
func (d *Daemon) execCommandGC() {
    for range time.Tick(5 * time.Minute) {
        var (
            cleaned          int
            liveExecCommands = d.containerExecIds()
        )
        for id, config := range d.execCommands.s {
            if config.canRemove {
                cleaned++
                d.execCommands.Delete(id)
            } else {
                if _, exists := liveExecCommands[id]; !exists {
                    config.canRemove = true
                }
            }
        }
        logrus.Debugf("clean %d unused exec commands", cleaned)
    }   
}

然后系统会调用下restore加载现有的container到内存中。接着我们的daemon就完成了。

我们来总结下daemon.NewDaemon。其做的事情就是初始化各个子模块,比如graph,net,exec等等。这里的初始化有两种,一种是直接得到一个子模块的struct,还有一种是获取一个driver,这里的driver可以认为是一个接口,然后初始化的时候会获取这个接口的一个实现。另外这里的初始化不仅仅是代码上的获取类这么简单,还会做很多物理上的操作,比如设置aufs的目录、建立sqlite db等等。在这些初始化完成后我们的daemon就拥有了强大的功能,从它入手就可以做很多东西了(它的每个属性都是我们初始化得到的重要结构体)。接着这里会调用下daemon的restore方法恢复系统上已有的container。

发表评论

电子邮件地址不会被公开。 必填项已用*标注

*