docs-v1/design_proposals/sync-auto-support.md
Currently skaffold does not support sync for files that are generated
during a build. For example when syncing java files to a container, one
would normally expect .class files to be sync'd, but skaffold is
really only aware of .java files in a build.
_smart_ described in sync-improvementsThe current sync system has the following problems:
./gradlew jib, but to update class files so the
system may sync them, one would only be required to run ./gradlew classes..java files for a build, in the sync case, we want
it to watch .java files, trigger a partial build, and sync .class files
so a remote server can pick up and reload the changes.To get close to the functionality we want, without modifying skaffold at all, a Dockerfile which depends on java build outputs could be used, like:
FROM openjdk:8
COPY build/dependencies/ /app/dependencies
COPY build/classes/java/main/ /app/classes
CMD ["java", "-cp", "/app/classes:/app/dependencies/*", "hello.Application"]
with a skaffold sync block that looks like:
sync:
manual:
- src: "build/classes/java/main/**/*.class"
dest: "/app/classes"
strip: "build/classes/java/main/"
A user's devloop then looks like this:
./gradlew classes copyDependenciesskaffold dev./gradlew classeswhich is far from ideal.
auto optionProvide users with an auto option, users that use auto should expect
the builder-sync to do the right thing and will not be required to do much
configuration.
auto will only work with builders that have implemented the auto spec. We
expect at least jib to do implement the spec.
build:
artifacts:
- image: ...
context: jib-project
jib: {}
sync:
auto: {}
Skaffold can expose an API that can accept a complex configuration on how skaffold should be doing synchronization.
The builder will talk to sync component by providing it with the following data
generated configs, each containing
command to generate files to syncdirect sync directives (src, dest) to execute without any script
executionSo maybe some datastructures like (I don't really know a lot of go, so assume this will be written in some consistent way eventually):
type AutoSync struct {
generated []Generated
direct []Direct
}
type Generated struct {
command []String
inputs []File
syncables []Syncables
}
type Direct struct {
syncables []Syncables
}
type Syncables struct {
src String
dest String
}
How a tool like Jib might surface the necessary information to Skaffold
I would expect to add a task like _jibSkaffoldSyncMap that will produce
json output for the skaffold-jib intergration to consume and forward to the sync
system. And example output might look like:
BEGIN JIB JSON: SYNCMAP/1
{
"generated": [
{
src: “target/classes”
dest: "app/classes",
},
{
src: “target/resources"
dest: "app/resources",
}
]
"direct": [
{
src: "src/main/extra1",
dest: "/"
},
{
src: "src/main/extra2",
dest: "/"
},
{
src: ".m2/some/dep/my-dep-SNAPSHOT.jar",
dest: "app/dependencies/my-dep-SNAPSHOT.jar"
}
}
}
Files in the generated section will trigger a partial rebuild of the container
(everything before containerization) while files in the direct section can
just be synchronized to the running container without a rebuild of anything.
Each builder implementing an auto builder should be able to decide when a sync
should be skipped and a rebuild done instead. In the jib case for instance, a
rebuild will be triggered if:
build.gradle, pom.xml, etc)What about files that have specific permissions?
Jib allows users to customize file permissions, for instance a file on the container can be configured to be 0x456 or something. One option instead of dealing with this, is to just make all sync'd files 777? Or we allow passthrough of permissions from the build system to the sync system.
Should we allow the user to configure the auto block?
Perhaps the user knows something that jib doesnt, and wants to ignore some files from synchronization. They might want to do:
sync:
auto:
ignored:
- "src/main/jib/myExtraFilesThatBreakADeployment"
[schemas/<version>.go] Add Auto to the schema under Sync
Before first build, initialize the sync state for builders in auto mode, this
sync state is saved in the builder's specific implementation of auto
On file change in dev mode:
if (files were deleted)
return REBUILD
if (changes were made to build def)
return REBUILD
lastSyncState = syncStates["this project"]
if (if all files changes are in lastSyncState.direct)
return SYNC{list of direct files}
newSyncState = buildAndCalculateNewSyncState("this project")
syncStates["this project"] = newSyncState
diff = diff(lastSyncState, newSyncState)
return SYNC{files in diff}
This can be a Kind based test, we don't need GCP secrets. Hence:
The following implementation had a few problems and that were potential deal breakers, this implementation is left in here for information purposes, no attempt was made to implement this. It has the following issues:
sync using tar which should be an implementation detailauto sync configuration
block. As skaffold takes care of fewer things, the user must be responsible
for implementing it.One option is to complete hand over the detection of container updates to the builder which would provide skaffold with a tar to synchronize.
A potential runthrough might look like:
tar to syncThis allows the builder implementation to make the sync decisions within it's own system.
For a potential consumer of this mechanism like jib, we would expose a task like
_jibCreateSyncTar which would so something like
REBUILD
SYNC: /path/to/sync.tar