Commit Graph

43 Commits

Author SHA1 Message Date
Mehran Kholdi
83e184c58a Remove redundant ARGs in dockerfile 2020-07-18 09:46:51 +04:30
Mehran Kholdi
77862b85e2 Support custom fsTypes
Test Plan:
- Deploy using `feature-fstype` image tag
- Create the following storage class:
```
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: btrfs-sc
parameters:
  fsType: btrfs
provisioner: rawfile.hamravesh.com
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
```
- Create and use a pvc backed by the new storage class
- Exec into the pod and verify that the mounted volume is a `btrfs` filesystem indeed

Reviewers: h.marvi, sina_rad, mhyousefi, bghadiri

Reviewed By: h.marvi, mhyousefi, bghadiri

Differential Revision: https://phab.hamravesh.ir/D833
2020-07-18 09:46:05 +04:30
Mehran Kholdi
0095c0e83a Implement metadata schema migration
Summary: Trying to add custom `fsType`s, new metadata fields need to be stored. We need a mechanism to migrate existing volume metadata.

Test Plan:
- Install the chart using an older image tag like `36fc480`
- Create and use a pvc
- Verify that the volume's metadata file, located at `/var/csi/rawfile/pvc-.../disk.meta` does not contain the `schema_version` field
- Upgrade the chart to use the image tag `feature-schema-migration`
- Wait until all node pods are upgraded
- Verify that the volume's metadata file contains the new `schema_version` field

Reviewers: bghadiri, h.marvi, mhyousefi, sina_rad

Reviewed By: bghadiri, h.marvi, mhyousefi

Differential Revision: https://phab.hamravesh.ir/D832
2020-07-18 09:41:24 +04:30
Mehran Kholdi
d606f8f064 [skip-e2e] Use slugified branch name as docker image tag 2020-07-13 21:18:09 +04:30
Mehran Kholdi
36fc480d28 [skip-e2e] Use openebs's docker image in helm chart 2020-07-13 20:05:40 +04:30
Mehran Kholdi
7a4d90a315 Run e2e tests in parallel when possible 2020-07-11 23:14:13 +04:30
Mehran Kholdi
ab407f3349 Setup CI: Build, run e2e tests, and push images to docker hub 2020-07-11 23:14:13 +04:30
Mehran Kholdi
16f92da467 Add "Motivation" section to README 2020-07-11 19:03:23 +04:30
fossabot
ee4d9ae660 Add license scan report and status
Signed off by: fossabot <badges@fossa.com>
2020-07-11 19:03:04 +04:30
Mehran Kholdi
8b1be18a15 Hotfix: Make side effects idempotent 2020-06-14 05:10:32 +04:30
Mehran Kholdi
d1c0d49cf0 Support online volume expansion
Summary:
Online volume expansion is a 2 phase process:

1. The backing storage, in this case the raw file, needs to be resized. (i.e. `truncate -s`)
2. The node should be notified, so that it can both refresh its device capacity (i.e. `losetup -c`) and resize the filesystem (`resize2fs`) accordingly.

Although in our case both steps could be performed on the node itself, for the sake of following the semantics of how volume expansion works, we perform step 1 from the controller, and step 2 from the node.

Also, the `external-resizer` component is added which watches for PVC size updates, and notifies the CSI controller about it.

Test Plan:
Setup:
- Deploy
- Create a rawfile-backed pvc, and attach a Deployment to it
- Keep an eye on `rawfile` pod logs in `kube-system` namespace to see if any errors pop out during all scenarios

Scenario 1:
- Increase the size of the pvc
- Exec into the pod and verify that the volume is resized indeed (using `df`)

Scenario 2:
- Decrease deployment's replica to 0
- Increase the size of the pvc. Wait for a couple of minutes.
- Increase deployment's replica to 1
- Exec into the pod and verify that the volume is resized indeed.

Reviewers: bghadiri, mhyousefi, h.marvi, sina_rad

Reviewed By: bghadiri, mhyousefi, sina_rad

Differential Revision: https://phab.hamravesh.ir/D817
2020-06-14 03:35:17 +04:30
Mehran Kholdi
be2cd1b72c Expose inode-related metrics
Test Plan: - Check `:9100/metrics` for existence of inode-related metrics

Reviewers: h.marvi, mhyousefi

Reviewed By: h.marvi

Differential Revision: https://phab.hamravesh.ir/D816
2020-06-14 03:33:33 +04:30
Mehran Kholdi
8e5cb5de78 Rewrite: Change the way we're exposing volume metrics
Summary:
Formerly, we were updating the metrics every 15 seconds. We were facing a couple of issues doing it manually:

- Outdated metrics in case of a one-time crash
- Metrics getting exposed for deleted PVs

Instead of fixing the bugs, I preferred to do it the right way. As per `python-prometheus` docs:

>  Sometimes it is not possible to directly instrument code, as it is not in your control. This requires you to proxy metrics from other systems. To do so you need to create a custom collector...

Test Plan:
- Deploy on a cluster with existing rawfile PVs
- Send request to `:9100/metrics` and assert that metrics are exposed
- Delete a PV, and assert that its metrics disappear

Reviewers: h.marvi, bghadiri, sina_rad, mhyousefi

Reviewed By: h.marvi, bghadiri, sina_rad

Differential Revision: https://phab.hamravesh.ir/D815
2020-06-14 03:31:21 +04:30
Mehran Kholdi
523ccd510f Use slim base image to reduce the resulting image size 2020-06-13 18:25:28 +04:30
kmova
fd7db09a0c chore(docs): add contributor guidelines
Signed-off-by: kmova <kiran.mova@mayadata.io>
2020-06-13 06:08:25 +00:00
Mehran Kholdi
5397774ce6 Publish under Apache License 2.0 2020-06-12 02:31:02 +04:30
Mehran Kholdi
b2c4b77911 Implement STAGE_UNSTAGE_VOLUME capability
Summary: Before this, we directly mounted the the rawfile on the mountpoint. In this revision the `STAGE_UNSTAGE_VOLUME` capability is implemented, meaning that the volume is first mounted to a staging path, and then `bind`-mounted to the actual path. This way we can free up loopback devices when they are not needed.

Test Plan:
- Create a pvc, and use it inside a pod
- Run `losetup -l` on the node running the pod, and assert the creation of a loop device
- Delete the pod, but not the pvc
- Run `losetup -l` on the same node, and assert the removal of the loop device

Reviewers: h.marvi, bghadiri

Differential Revision: https://phab.hamravesh.ir/D806
2020-05-31 13:23:10 +04:30
Mehran Kholdi
f557aef8ec Pin base image version
We rely on the output format of some command line tools, and hence
need to know we're getting the same version of them.
2020-05-31 13:20:15 +04:30
Mehran Kholdi
46bd3565a0 Use immutable tags for running tasks
This way we can run tasks with `IfNotPresent` and be sure it works as intended.
2020-05-29 21:04:40 +04:30
Mehran Kholdi
0ab1e393fc Fix typo 2020-05-29 21:04:20 +04:30
Mehran Kholdi
a51305e878 Push official images to dockerhub 2020-05-29 21:04:20 +04:30
Mehran Kholdi
b49f8304f5 Add component label to controller service 2020-05-03 19:28:55 +04:30
Mehran Kholdi
87dd918a74 Update ServiceMonitor to chart 2020-04-28 22:54:38 +04:30
Mehran Kholdi
a6a859fc0e Add installation and usage instructions to readme 2020-04-28 22:54:38 +04:30
Mehran Kholdi
a2fdcb9756 Deploy controller using statefulset instead of deployment
So that no two controllers are running simultaneously.
2020-04-28 22:54:38 +04:30
Mehran Kholdi
420b0f4ff7 Create helm chart 2020-04-28 22:54:38 +04:30
Mehran Kholdi
336f72a3c9 Cleanup manifests 2020-04-26 02:02:00 +04:30
Mehran Kholdi
974ad321a9 Hardcode block size as 512 instead of using blksize!
See: https://stackoverflow.com/a/3212102/585678
See: https://stackoverflow.com/a/12952830/585678
2020-04-26 02:02:00 +04:30
Mehran Kholdi
9deaefc203 Implement basic metrics 2020-04-26 02:02:00 +04:30
Mehran Kholdi
c9da83b19b Setup loopback devices using Direct I/O 2020-04-26 02:01:42 +04:30
Mehran Kholdi
dcc2210b3c Handle attaching loop devices instead of handing it to mount
* So that we can pass extra loop options like Direct I/O
* To workaround an issue with loop device files not getting
automatically created
2020-04-26 02:01:42 +04:30
Mehran Kholdi
c500df8ca2 Delete task pods upon successful completion 2020-04-26 01:37:23 +04:30
Mehran Kholdi
4ec6acf731 Store metadata file 2020-04-26 01:37:23 +04:30
Mehran Kholdi
373c43fa0e Add simple validations to CreateVolume request 2020-04-26 01:37:23 +04:30
Mehran Kholdi
9de82a9b47 Allocate new volumes remotely 2020-04-26 01:37:23 +04:30
Mehran Kholdi
73a618228b Temp: Disable python buffering
So that we can debug based on stdout
2020-04-26 01:37:23 +04:30
Mehran Kholdi
da2856237b Switch from alpine base to debian
So that python requirements could be installed from wheels.
2020-04-26 01:37:23 +04:30
Mehran Kholdi
0dec2f269b Add README 2020-04-26 01:37:22 +04:30
Mehran Kholdi
18abfa1ec4 Configure CI 2020-04-24 19:35:37 +04:30
Mehran Kholdi
6f23dbe067 Initial PoC of the csi driver 2020-04-24 19:35:37 +04:30
Mehran Kholdi
98139d428e Autogen csi grpc interface
Proto taken from:
https://github.com/container-storage-interface/spec/blob/master/csi.proto

Compiled using:
`python -m grpc_tools.protoc -I./protos --python_out=./csi/ --grpc_python_out=./csi/ protos/csi.proto`
2020-04-24 00:08:36 +04:30
Mehran Kholdi
6ac237abb8 Autogen python gitignore 2020-04-23 04:18:53 +04:30
Mehran Kholdi
7df7dc3ee5 Initial commit 2020-04-23 04:13:33 +04:30