Skip to content

feat(v2): tensor column mode#886

Merged
samsja merged 53 commits intofeat-rewrite-v2from
feat-tensor-column-mode
Dec 6, 2022
Merged

feat(v2): tensor column mode#886
samsja merged 53 commits intofeat-rewrite-v2from
feat-tensor-column-mode

Conversation

@samsja
Copy link
Member

@samsja samsja commented Dec 2, 2022 •
edited
Loading

Context

introducing the stacked mode for faster computation and saving memory when dealing with tensor when needed:

class Image(Document):
    tensor: TorchTensor[3, 224, 224]

batch = DocumentArray[Image](
    [Image(tensor=torch.zeros(3, 224, 224)) for _ in range(10)]
)

batch.stack()
assert (batch.tensor == torch.zeros(10, 3, 224, 224)).all()
assert batch.is_stack()
batch.unstack()

We expose as well a context manager

with batch.stacked_mode():
    ...

with batch.unstacked_mode():
   ...

@samsja samsja marked this pull request as draft December 2, 2022 10:26
@samsja samsja changed the title Feat tensor column mode feat(v2): tensor column mode Dec 2, 2022
@github-actions github-actions bot added size/l and removed size/m labels Dec 5, 2022
@samsja samsja force-pushed the feat-tensor-column-mode branch from fe10440 to 31d85d8 Compare December 5, 2022 15:27
@samsja samsja requested a review from alexcg1 December 5, 2022 16:40
@samsja samsja marked this pull request as ready for review December 5, 2022 16:41
@samsja samsja force-pushed the feat-tensor-column-mode branch from 915406d to 342a726 Compare December 5, 2022 16:41
JohannesMessner
JohannesMessner requested changes Dec 5, 2022
View reviewed changes
@samsja samsja force-pushed the feat-tensor-column-mode branch 3 times, most recently from 63c74c7 to 62671f3 Compare December 6, 2022 10:01
JohannesMessner
JohannesMessner requested changes Dec 6, 2022
View reviewed changes
alexcg1
alexcg1 suggested changes Dec 6, 2022
View reviewed changes
docarray/array/array.py


**Unstacked mode (default)**:
In this case a DocumentArray is a list of Documents and each Document owns its data.
Copy link
Contributor

@alexcg1 alexcg1 Dec 6, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does it mean to for a Document to "own" data?

Copy link
Member Author

@samsja samsja Dec 6, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it means that the data is bound to the lifecycle of the Document. If the Document go out of scope the data as well

Copy link
Contributor

@alexcg1 alexcg1 Dec 6, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it also be accurate to say the Document stores the data?

Copy link
Member Author

@samsja samsja Dec 6, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think here owning the the good term from a computer science point of view

@samsja samsja force-pushed the feat-tensor-column-mode branch from 78b7968 to b6770d2 Compare December 6, 2022 12:58
samsja added 10 commits December 6, 2022 13:59
@samsja
fix: inherit from cls in da
a574d84 
Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
@samsja
fix: move class atribute to top torch tensor
05b90e9 
Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
@samsja
feat: add stack and unstack mod
df5fe28 
Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
@samsja
feat: add blocker for stack mod
915cbf5 
Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
@samsja
feat: add blocker for stack mod
fd7d8a7 
Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
@samsja
fix: fix issue with dict iterationg
95316af 
Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
@samsja
refactor: init column in stacked mode entry and not init
70d6430 
Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
@samsja
fix: fix typo
b290040 
Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
@samsja
fix: fix typo
ac26cda 
Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
@samsja
feat: add column mode for tensor
f50c27b 
Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
samsja and others added 15 commits December 6, 2022 14:01
@samsja
feat: add warning in list like funciton
bcaad73 
Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
@samsja
fix: fix docstring column fields
ce34689 
Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
@samsja
feat: add warning in list like funciton
1f57835 
Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
@samsja
feat: apply jihannes suggestion
a6abd5e 
Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
@samsja @JohannesMessner
feat: apply aleaddine suggestion
834fada 
Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com>
Signed-off-by: samsja <55492238+samsja@users.noreply.github.com>
Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
@samsja
feat: apply jihannes suggestion
bc76613 
Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
@samsja
feat: apply jihannes suggestion
17802b3 
Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
@samsja
feat: apply jihannes suggestion
052288f 
Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
@samsja
fix: docstring for blocked method
b0dd31d 
Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
@samsja
feat: add better docstring for context manager
5fd7864 
Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
@samsja
fix: mypy
c1bfa6f 
Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
@samsja @alexcg1
feat: apply alex suggestion
9a75e80 
Co-authored-by: Alex Cureton-Griffiths <alexcg1@users.noreply.github.com>
Signed-off-by: samsja <55492238+samsja@users.noreply.github.com>
Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
@samsja @alexcg1
feat: apply alex suggestion
d0d75c0 
Co-authored-by: Alex Cureton-Griffiths <alexcg1@users.noreply.github.com>
Signed-off-by: samsja <55492238+samsja@users.noreply.github.com>
Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
@samsja
feat: apply alex suggestion
e0fec6b 
Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
@samsja
fix: mypy
0b3f538 
Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
@samsja samsja force-pushed the feat-tensor-column-mode branch from b6770d2 to 0b3f538 Compare December 6, 2022 13:01
samsja added 2 commits December 6, 2022 14:02
@samsja
fix: rebase
228277d 
Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
@samsja
fix: rebase
6d07939 
Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
alexcg1
alexcg1 suggested changes Dec 6, 2022
View reviewed changes
Copy link
Contributor

@alexcg1 alexcg1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I missed a few things last time round 😅

docarray/array/array.py


**Unstacked mode (default)**:
In this case a DocumentArray is a list of Documents and each Document owns its data.
Copy link
Contributor

@alexcg1 alexcg1 Dec 6, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it also be accurate to say the Document stores the data?

samsja and others added 4 commits December 6, 2022 15:37
@samsja
refactor: fix problem
65ba015 
Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
@samsja @alexcg1
feat: apply alex suggestion
baa7f53 
Co-authored-by: Alex Cureton-Griffiths <alexcg1@users.noreply.github.com>
Signed-off-by: samsja <55492238+samsja@users.noreply.github.com>
@samsja @alexcg1
feat: apply alex suggestion
d7c0458 
Co-authored-by: Alex Cureton-Griffiths <alexcg1@users.noreply.github.com>
Signed-off-by: samsja <55492238+samsja@users.noreply.github.com>
@samsja
fix: apply black to suggestion
0049d66 
Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
alexcg1
alexcg1 approved these changes Dec 6, 2022
View reviewed changes
Copy link
Contributor

@alexcg1 alexcg1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

@samsja
fix: update docstring
1286876 
Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
@github-actions
Copy link

github-actions bot commented Dec 6, 2022

📝 Docs are deployed on https://ft-feat-tensor-column-mode--jina-docs.netlify.app 🎉

@samsja samsja merged commit ae03664 into feat-rewrite-v2 Dec 6, 2022
@samsja samsja deleted the feat-tensor-column-mode branch December 6, 2022 15:55
@JohannesMessner JohannesMessner mentioned this pull request Dec 13, 2022
Closed
47 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@JohannesMessner JohannesMessner JohannesMessner approved these changes

+1 more reviewer

@alexcg1 alexcg1 alexcg1 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

area/core area/testing area/typing component/array size/l

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@samsja @alexcg1 @JohannesMessner