Commit Graph

59 Commits

Author SHA1 Message Date
Hanieh Marvi
ba7f4c1b7f Remove requests from tasks
So pods do not stay in pending state because of lack of resources.
2021-02-13 02:03:04 +03:30
Hanieh Marvi
8424536588 Set resources for sidecar container 2021-02-13 02:03:04 +03:30
Mehran Kholdi
ab50217ea5 Release 0.4.2 2021-01-16 04:01:22 +03:30
Mehran Kholdi
b4faf9d7cb Expose volume metrics through gRPC calls rather than metrics endpoint 2021-01-16 03:58:08 +03:30
Mehran Kholdi
c58dd14bf7 Extract blockdevice-to-filesystem logic from rawfile servicer
Summary: So that it's possible to use it with any other blockdevice provider.

Test Plan: N/A

Reviewers: sina_rad, h.marvi, mhyousefi, s.afshari

Differential Revision: https://phab.hamravesh.ir/D870
2021-01-16 03:58:08 +03:30
Mehran Kholdi
01a35354b6 Fix a bug where broken symlinks where not being cleaned up
See: https://docs.python.org/3/library/pathlib.html#pathlib.Path.exists
"Note If the path points to a symlink, exists() returns whether the symlink points to an existing file or directory."
2021-01-16 03:45:09 +03:30
Mehran Kholdi
c2110108cb Change conditions upon which e2e test are run 2020-11-28 04:50:30 +03:30
Mehran Kholdi
9bafb101ac Remove liveness probes 2020-11-28 04:50:11 +03:30
Mehran Kholdi
05c661165f Fix ci setup script
So that it does not explicitly depend on travis
2020-11-08 01:46:08 +03:30
Mehran Kholdi
b88fd0cfdf Release 0.4.1 2020-09-11 20:45:17 +04:30
Mehran Kholdi
23c7912977 Update chart's default values 2020-09-11 20:44:56 +04:30
Mehran Kholdi
a2cf384d4f Make logs less noisy 2020-09-11 20:44:40 +04:30
Mehran Kholdi
6fde8e0271 Update external csi sidecar containers 2020-09-11 20:44:29 +04:30
Mehran Kholdi
848d87453f Change default provisioner name from rawfile.hamravesh.com to rawfile.csi.openebs.io 2020-08-15 01:36:05 +04:30
Mehran Kholdi
13a16e70f6 Use semantic versioning 2020-08-14 20:11:07 +04:30
Mehran Kholdi
c895312131 Implement GET_VOLUME_STATS capability 2020-07-18 09:50:05 +04:30
Mehran Kholdi
83e184c58a Remove redundant ARGs in dockerfile 2020-07-18 09:46:51 +04:30
Mehran Kholdi
77862b85e2 Support custom fsTypes
Test Plan:
- Deploy using `feature-fstype` image tag
- Create the following storage class:
```
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: btrfs-sc
parameters:
  fsType: btrfs
provisioner: rawfile.hamravesh.com
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
```
- Create and use a pvc backed by the new storage class
- Exec into the pod and verify that the mounted volume is a `btrfs` filesystem indeed

Reviewers: h.marvi, sina_rad, mhyousefi, bghadiri

Reviewed By: h.marvi, mhyousefi, bghadiri

Differential Revision: https://phab.hamravesh.ir/D833
2020-07-18 09:46:05 +04:30
Mehran Kholdi
0095c0e83a Implement metadata schema migration
Summary: Trying to add custom `fsType`s, new metadata fields need to be stored. We need a mechanism to migrate existing volume metadata.

Test Plan:
- Install the chart using an older image tag like `36fc480`
- Create and use a pvc
- Verify that the volume's metadata file, located at `/var/csi/rawfile/pvc-.../disk.meta` does not contain the `schema_version` field
- Upgrade the chart to use the image tag `feature-schema-migration`
- Wait until all node pods are upgraded
- Verify that the volume's metadata file contains the new `schema_version` field

Reviewers: bghadiri, h.marvi, mhyousefi, sina_rad

Reviewed By: bghadiri, h.marvi, mhyousefi

Differential Revision: https://phab.hamravesh.ir/D832
2020-07-18 09:41:24 +04:30
Mehran Kholdi
d606f8f064 [skip-e2e] Use slugified branch name as docker image tag 2020-07-13 21:18:09 +04:30
Mehran Kholdi
36fc480d28 [skip-e2e] Use openebs's docker image in helm chart 2020-07-13 20:05:40 +04:30
Mehran Kholdi
7a4d90a315 Run e2e tests in parallel when possible 2020-07-11 23:14:13 +04:30
Mehran Kholdi
ab407f3349 Setup CI: Build, run e2e tests, and push images to docker hub 2020-07-11 23:14:13 +04:30
Mehran Kholdi
16f92da467 Add "Motivation" section to README 2020-07-11 19:03:23 +04:30
fossabot
ee4d9ae660 Add license scan report and status
Signed off by: fossabot <badges@fossa.com>
2020-07-11 19:03:04 +04:30
Mehran Kholdi
8b1be18a15 Hotfix: Make side effects idempotent 2020-06-14 05:10:32 +04:30
Mehran Kholdi
d1c0d49cf0 Support online volume expansion
Summary:
Online volume expansion is a 2 phase process:

1. The backing storage, in this case the raw file, needs to be resized. (i.e. `truncate -s`)
2. The node should be notified, so that it can both refresh its device capacity (i.e. `losetup -c`) and resize the filesystem (`resize2fs`) accordingly.

Although in our case both steps could be performed on the node itself, for the sake of following the semantics of how volume expansion works, we perform step 1 from the controller, and step 2 from the node.

Also, the `external-resizer` component is added which watches for PVC size updates, and notifies the CSI controller about it.

Test Plan:
Setup:
- Deploy
- Create a rawfile-backed pvc, and attach a Deployment to it
- Keep an eye on `rawfile` pod logs in `kube-system` namespace to see if any errors pop out during all scenarios

Scenario 1:
- Increase the size of the pvc
- Exec into the pod and verify that the volume is resized indeed (using `df`)

Scenario 2:
- Decrease deployment's replica to 0
- Increase the size of the pvc. Wait for a couple of minutes.
- Increase deployment's replica to 1
- Exec into the pod and verify that the volume is resized indeed.

Reviewers: bghadiri, mhyousefi, h.marvi, sina_rad

Reviewed By: bghadiri, mhyousefi, sina_rad

Differential Revision: https://phab.hamravesh.ir/D817
2020-06-14 03:35:17 +04:30
Mehran Kholdi
be2cd1b72c Expose inode-related metrics
Test Plan: - Check `:9100/metrics` for existence of inode-related metrics

Reviewers: h.marvi, mhyousefi

Reviewed By: h.marvi

Differential Revision: https://phab.hamravesh.ir/D816
2020-06-14 03:33:33 +04:30
Mehran Kholdi
8e5cb5de78 Rewrite: Change the way we're exposing volume metrics
Summary:
Formerly, we were updating the metrics every 15 seconds. We were facing a couple of issues doing it manually:

- Outdated metrics in case of a one-time crash
- Metrics getting exposed for deleted PVs

Instead of fixing the bugs, I preferred to do it the right way. As per `python-prometheus` docs:

>  Sometimes it is not possible to directly instrument code, as it is not in your control. This requires you to proxy metrics from other systems. To do so you need to create a custom collector...

Test Plan:
- Deploy on a cluster with existing rawfile PVs
- Send request to `:9100/metrics` and assert that metrics are exposed
- Delete a PV, and assert that its metrics disappear

Reviewers: h.marvi, bghadiri, sina_rad, mhyousefi

Reviewed By: h.marvi, bghadiri, sina_rad

Differential Revision: https://phab.hamravesh.ir/D815
2020-06-14 03:31:21 +04:30
Mehran Kholdi
523ccd510f Use slim base image to reduce the resulting image size 2020-06-13 18:25:28 +04:30
kmova
fd7db09a0c chore(docs): add contributor guidelines
Signed-off-by: kmova <kiran.mova@mayadata.io>
2020-06-13 06:08:25 +00:00
Mehran Kholdi
5397774ce6 Publish under Apache License 2.0 2020-06-12 02:31:02 +04:30
Mehran Kholdi
b2c4b77911 Implement STAGE_UNSTAGE_VOLUME capability
Summary: Before this, we directly mounted the the rawfile on the mountpoint. In this revision the `STAGE_UNSTAGE_VOLUME` capability is implemented, meaning that the volume is first mounted to a staging path, and then `bind`-mounted to the actual path. This way we can free up loopback devices when they are not needed.

Test Plan:
- Create a pvc, and use it inside a pod
- Run `losetup -l` on the node running the pod, and assert the creation of a loop device
- Delete the pod, but not the pvc
- Run `losetup -l` on the same node, and assert the removal of the loop device

Reviewers: h.marvi, bghadiri

Differential Revision: https://phab.hamravesh.ir/D806
2020-05-31 13:23:10 +04:30
Mehran Kholdi
f557aef8ec Pin base image version
We rely on the output format of some command line tools, and hence
need to know we're getting the same version of them.
2020-05-31 13:20:15 +04:30
Mehran Kholdi
46bd3565a0 Use immutable tags for running tasks
This way we can run tasks with `IfNotPresent` and be sure it works as intended.
2020-05-29 21:04:40 +04:30
Mehran Kholdi
0ab1e393fc Fix typo 2020-05-29 21:04:20 +04:30
Mehran Kholdi
a51305e878 Push official images to dockerhub 2020-05-29 21:04:20 +04:30
Mehran Kholdi
b49f8304f5 Add component label to controller service 2020-05-03 19:28:55 +04:30
Mehran Kholdi
87dd918a74 Update ServiceMonitor to chart 2020-04-28 22:54:38 +04:30
Mehran Kholdi
a6a859fc0e Add installation and usage instructions to readme 2020-04-28 22:54:38 +04:30
Mehran Kholdi
a2fdcb9756 Deploy controller using statefulset instead of deployment
So that no two controllers are running simultaneously.
2020-04-28 22:54:38 +04:30
Mehran Kholdi
420b0f4ff7 Create helm chart 2020-04-28 22:54:38 +04:30
Mehran Kholdi
336f72a3c9 Cleanup manifests 2020-04-26 02:02:00 +04:30
Mehran Kholdi
974ad321a9 Hardcode block size as 512 instead of using blksize!
See: https://stackoverflow.com/a/3212102/585678
See: https://stackoverflow.com/a/12952830/585678
2020-04-26 02:02:00 +04:30
Mehran Kholdi
9deaefc203 Implement basic metrics 2020-04-26 02:02:00 +04:30
Mehran Kholdi
c9da83b19b Setup loopback devices using Direct I/O 2020-04-26 02:01:42 +04:30
Mehran Kholdi
dcc2210b3c Handle attaching loop devices instead of handing it to mount
* So that we can pass extra loop options like Direct I/O
* To workaround an issue with loop device files not getting
automatically created
2020-04-26 02:01:42 +04:30
Mehran Kholdi
c500df8ca2 Delete task pods upon successful completion 2020-04-26 01:37:23 +04:30
Mehran Kholdi
4ec6acf731 Store metadata file 2020-04-26 01:37:23 +04:30
Mehran Kholdi
373c43fa0e Add simple validations to CreateVolume request 2020-04-26 01:37:23 +04:30